Jul 17 18:33:09.930897 ip-10-0-138-130 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 18:33:10.378205 ip-10-0-138-130 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:33:10.378205 ip-10-0-138-130 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 18:33:10.378205 ip-10-0-138-130 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:33:10.378205 ip-10-0-138-130 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 18:33:10.378205 ip-10-0-138-130 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:33:10.381181 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.381088    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 18:33:10.387078 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387051    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:10.387078 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387073    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:10.387078 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387080    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:10.387078 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387085    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387089    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387093    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387097    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387100    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387104    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387108    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387112    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387115    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387119    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387123    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387127    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387130    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387135    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387138    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387142    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387146    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387149    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387153    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387157    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:10.387336 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387160    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387165    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387168    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387173    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387177    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387181    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387184    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387188    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387192    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387196    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387200    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387204    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387208    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387213    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387217    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387222    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387226    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387230    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387234    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387238    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:10.388173 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387243    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387247    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387251    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387255    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387259    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387264    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387268    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387275    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387282    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387287    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387292    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387297    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387349    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387360    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387365    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387370    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387375    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387382    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387388    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:10.389075 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387393    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387397    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387401    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387405    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387409    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387414    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387418    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387424    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387428    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387432    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387436    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387440    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387445    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387449    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387453    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387459    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387463    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387467    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387473    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387477    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:10.389883 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387481    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387487    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387491    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387496    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387500    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387504    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387508    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387513    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387517    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387522    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387526    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387530    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387556    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387562    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387566    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387573    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.387579    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388800    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388811    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:10.390745 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388816    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388821    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388825    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388829    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388834    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388838    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388843    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388847    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388851    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388855    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388859    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388864    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388868    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388872    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388877    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388882    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388886    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388891    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388898    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388904    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:10.391555 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388908    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388912    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388917    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388921    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388926    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388930    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388934    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388938    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388942    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388946    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388951    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388955    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388961    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388965    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388970    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388974    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388978    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388982    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388986    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388990    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:10.392251 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388994    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.388998    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389003    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389007    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389011    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389015    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389022    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389028    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389033    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389037    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389042    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389046    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389051    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389056    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389060    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389065    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389070    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389074    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389078    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:10.392777 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389082    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389086    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389093    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389097    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389102    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389106    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389110    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389115    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389119    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389126    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389131    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389136    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389141    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389145    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389149    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389153    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389157    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389162    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389166    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:10.393333 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389170    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389174    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389178    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389183    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389187    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389192    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389197    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389201    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389205    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389210    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389214    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389218    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389222    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389225    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389230    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389234    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389239    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389243    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389248    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389902    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:10.393900 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389912    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389916    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389921    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389925    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389929    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389933    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389938    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389942    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389946    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389950    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389954    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389958    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389963    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389968    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389975    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389981    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389985    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389989    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389993    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.389997    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:10.394607 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390001    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390006    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390010    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390013    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390018    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390022    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390026    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390030    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390036    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390040    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390045    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390049    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390054    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390059    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390063    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390067    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390071    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390075    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390079    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:10.395109 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390085    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390092    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390097    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390102    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390106    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390111    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390115    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390119    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390124    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390129    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390133    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390138    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390142    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390146    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390152    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390157    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390162    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390166    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390174    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:10.395663 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390179    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390183    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390188    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390192    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390196    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390201    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390208    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390213    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390217    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390221    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390226    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390230    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390234    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390238    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390242    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390246    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390251    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390255    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390259    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390263    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:10.396197 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390268    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390272    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390276    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390281    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390285    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390290    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390294    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390298    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390301    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390305    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390309    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390316    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390319    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390324    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390328    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390332    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390336    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390340    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390344    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.390348    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390463    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 18:33:10.396729 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390474    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390485    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390492    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390499    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390504    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390511    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390517    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390522    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390527    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390550    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390556    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390562    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390566    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390571    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390576    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390581    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390587    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390592    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390597    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390602    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390607    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390612    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390621    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 18:33:10.397233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390626    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390630    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390637    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390642    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390646    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390652    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390657    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390661    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390668    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390673    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390678    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390682    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390687    2563 flags.go:64] FLAG: --enable-server="true"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390692    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390699    2563 flags.go:64] FLAG: --event-burst="100"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390704    2563 flags.go:64] FLAG: --event-qps="50"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390708    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390714    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390719    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390726    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390730    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390735    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390740    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390744    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390749    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 18:33:10.397943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390754    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390758    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390763    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390767    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390772    2563 flags.go:64] FLAG: --feature-gates=""
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390778    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390785    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390790    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390803    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390808    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390813    2563 flags.go:64] FLAG: --help="false"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390818    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390822    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390827    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390832    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390838    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390844    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390848    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390853    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390857    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390861    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390866    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390871    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390876    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 17 18:33:10.398592 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390883    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390887    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390893    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390897    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390901    2563 flags.go:64] FLAG: --lock-file=""
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390906    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390910    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390915    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390925    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390930    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390935    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390939    2563 flags.go:64] FLAG: --logging-format="text"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390944    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390949    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390956    2563 flags.go:64] FLAG: --manifest-url=""
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390960    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390968    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390972    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390979    2563 flags.go:64] FLAG: --max-pods="110"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390984    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390989    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390993    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.390999    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391003    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391008    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 18:33:10.399253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391012    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391024    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391029    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391034    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391039    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391044    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391052    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391059    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391064    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391068    2563 flags.go:64] FLAG: --port="10250"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391073    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391077    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-095b6a48525a4a63a"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391083    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391088    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391092    2563 flags.go:64] FLAG: --register-node="true"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391097    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391103    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391108    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391113    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391117    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391131    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391136    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391143    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391147    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391152    2563 flags.go:64] FLAG: --runonce="false"
Jul 17 18:33:10.399903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391157    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391162    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391168    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391172    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391178    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391183    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391187    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391193    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391198    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391203    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391207    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391212    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391217    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391221    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391227    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391236    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391241    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391246    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391252    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391256    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391261    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391265    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391270    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391274    2563 flags.go:64] FLAG: --v="2"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391281    2563 flags.go:64] FLAG: --version="false"
Jul 17 18:33:10.400506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391288    2563 flags.go:64] FLAG: --vmodule=""
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391294    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.391299    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391492    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391502    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391506    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391511    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391516    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391520    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391524    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391528    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391550    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391554    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391558    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391562    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391566    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391570    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391574    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391578    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391582    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:10.401174 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391586    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391597    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391602    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391606    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391610    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391615    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391619    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391624    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391628    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391631    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391635    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391640    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391644    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391649    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391654    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391659    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391664    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391668    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391673    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391677    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:10.401678 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391681    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391685    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391689    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391693    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391697    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391702    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391706    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391710    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391714    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391721    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391727    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391731    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391738    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391745    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391751    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391756    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391761    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391766    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:10.402172 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391770    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391775    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391779    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391783    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391787    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391791    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391795    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391800    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391805    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391810    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391817    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391821    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391825    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391830    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391834    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391838    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391842    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391846    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391850    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391854    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:10.402628 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391858    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391862    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391866    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391870    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391875    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391879    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391883    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391889    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391893    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391897    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391902    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391906    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391910    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391915    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391918    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391923    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391927    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391931    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391935    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391939    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:10.403116 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391943    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391951    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391957    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.391962    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.392991    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.400620    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.400638    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.400660    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.400666    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400736    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400741    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400744    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400748    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400752    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:10.403623 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400755    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400758    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400761    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400764    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400767    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400770    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400772    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400775    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400779    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400783    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400786    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400789    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400792    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400796    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400798    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400801    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400804    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400806    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400809    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:10.403979 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400815    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400818    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400822    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400825    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400828    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400833    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400835    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400838    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400841    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400844    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400847    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400849    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400852    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400855    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400857    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400859    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400862    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400864    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400866    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400869    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:10.404529 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400871    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400873    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400876    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400879    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400881    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400883    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400886    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400888    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400891    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400893    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400896    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400898    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400900    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400903    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400906    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400908    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400911    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400914    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400917    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400919    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:10.405054 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400922    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400924    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400926    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400929    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400931    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400934    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400936    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400939    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400941    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400943    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400946    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400948    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400951    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400953    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400956    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400958    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400960    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400963    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400965    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400968    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:10.405548 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400970    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400972    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400975    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400978    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400980    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400983    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400989    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400991    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400994    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400996    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.400999    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401001    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401004    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401006    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401009    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:10.406044 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.401014    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401170    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401174    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401178    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401181    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401184    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401187    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401189    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401192    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401195    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401197    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401200    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401203    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401207    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401210    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401213    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401216    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401219    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401221    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:10.406415 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401225    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401227    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401231    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401234    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401237    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401241    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401244    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401247    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401250    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401252    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401256    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401259    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401262    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401264    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401267    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401270    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401272    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401275    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401277    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401280    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:10.406898 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401282    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401285    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401287    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401290    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401292    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401294    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401297    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401299    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401301    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401304    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401306    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401308    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401311    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401315    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401318    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401321    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401323    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401326    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401329    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:10.407378 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401331    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401334    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401337    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401340    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401342    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401344    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401347    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401349    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401352    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401354    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401356    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401359    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401361    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401364    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401366    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401368    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401371    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401373    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401376    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401378    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:10.407856 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401381    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401383    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401385    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401388    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401391    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401393    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401395    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401397    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401400    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401403    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401405    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401408    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401410    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401413    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401416    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401418    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401420    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401422    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401425    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401427    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:10.408354 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401430    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:10.408936 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:10.401432    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:10.408936 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.401436    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:33:10.408936 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.401577    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 18:33:10.408936 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.403703    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 18:33:10.408936 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.405786    2563 server.go:1019] "Starting client certificate rotation"
Jul 17 18:33:10.408936 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.405886    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 18:33:10.408936 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.405925    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 18:33:10.408936 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.408876    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 18:33:10.410388 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.410370    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 18:33:10.420571 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.420526    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 17 18:33:10.422635 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.422621    2563 log.go:25] "Validated CRI v1 image API"
Jul 17 18:33:10.423476 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.423455    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 18:33:10.428409 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.428385    2563 fs.go:135] Filesystem UUIDs: map[3278ed27-a710-4178-9944-a6465ad0b052:/dev/nvme0n1p4 69a84b75-f875-41ab-a07d-ae8a2136c73c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 17 18:33:10.428505 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.428409    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 18:33:10.433568 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.433432    2563 manager.go:216] Machine: {Timestamp:2026-07-17 18:33:10.432921025 +0000 UTC m=+0.377330605 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112179 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2656886612beddd1254258559b7d53 SystemUUID:ec265688-6612-bedd-d125-4258559b7d53 BootID:2c326519-b76a-4edb-aa8b-48973868d0e0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:b5:41:19:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:b5:41:19:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:83:4e:51:72:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 18:33:10.433568 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.433560    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 18:33:10.433666 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.433649    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 18:33:10.436885 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.436848    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 18:33:10.437043 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.436891    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 18:33:10.438825 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.438812    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 18:33:10.438864 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.438827    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 18:33:10.438864 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.438835    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 18:33:10.439725 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.439713    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 18:33:10.441946 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.441934    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 18:33:10.445124 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.445111    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 18:33:10.445264 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.445254    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 18:33:10.446472 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.446462    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 18:33:10.446512 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.446481    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 18:33:10.446512 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.446494    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 18:33:10.446512 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.446503    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 17 18:33:10.446512 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.446513    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 18:33:10.447007 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.446992    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 18:33:10.447055 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.447014    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 18:33:10.450004 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.449986    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 18:33:10.451410 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.451396    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 18:33:10.451453 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.451427    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 18:33:10.452557 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452530    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 18:33:10.452619 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452569    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 18:33:10.452619 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452583    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 18:33:10.452619 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452594    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 18:33:10.452619 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452607    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 18:33:10.452619 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452615    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 18:33:10.452767 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452627    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 18:33:10.452767 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452633    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 18:33:10.452767 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452643    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 18:33:10.452767 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452650    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 18:33:10.452767 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452670    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 18:33:10.452767 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.452676    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 18:33:10.453691 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.453681    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 18:33:10.453691 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.453691    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 18:33:10.455630 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.455609    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 18:33:10.457186 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.457169    2563 server.go:1268] "Started kubelet"
Jul 17 18:33:10.457342 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.457267    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 18:33:10.457388 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.457376    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 18:33:10.457824 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.457798    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 18:33:10.458119 ip-10-0-138-130 systemd[1]: Started Kubernetes Kubelet.
Jul 17 18:33:10.458644 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.458633    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 18:33:10.461105 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.461082    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 18:33:10.463177 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.463150    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 18:33:10.463835 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.463814    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 18:33:10.463991 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.463974    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-130.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 18:33:10.464217 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.464200    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 18:33:10.465520 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.464435    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-130.ec2.internal.18c327649d3db815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-130.ec2.internal,UID:ip-10-0-138-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-130.ec2.internal,},FirstTimestamp:2026-07-17 18:33:10.457128981 +0000 UTC m=+0.401538561,LastTimestamp:2026-07-17 18:33:10.457128981 +0000 UTC m=+0.401538561,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-130.ec2.internal,}"
Jul 17 18:33:10.468647 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.468621    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 18:33:10.468736 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.468626    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 18:33:10.468821 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.468801    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:10.469380 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469361    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 18:33:10.469380 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469365    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 18:33:10.469524 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469391    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 18:33:10.469524 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469487    2563 factory.go:55] Registering systemd factory
Jul 17 18:33:10.469524 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469503    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 17 18:33:10.469524 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469504    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 18:33:10.469524 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469513    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 18:33:10.469798 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469782    2563 factory.go:153] Registering CRI-O factory
Jul 17 18:33:10.469835 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469803    2563 factory.go:223] Registration of the crio container factory successfully
Jul 17 18:33:10.469883 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469862    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 18:33:10.469927 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469887    2563 factory.go:103] Registering Raw factory
Jul 17 18:33:10.469927 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.469902    2563 manager.go:1196] Started watching for new ooms in manager
Jul 17 18:33:10.470506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.470491    2563 manager.go:349] Starting recovery of all containers
Jul 17 18:33:10.471557 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.471463    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 18:33:10.471741 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.471599    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 18:33:10.480402 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.480384    2563 manager.go:354] Recovery completed
Jul 17 18:33:10.481736 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.481705    2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 17 18:33:10.484923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.484909    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:10.487072 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.487050    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:10.487131 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.487084    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:10.487131 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.487095    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:10.487708 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.487692    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:10.487763 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.487697    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 18:33:10.487763 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.487754    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 18:33:10.487815 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.487773    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 18:33:10.488794 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.488780    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:10.488859 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.488808    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:10.488859 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.488821    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:10.489295 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.489283    2563 policy_none.go:49] "None policy: Start"
Jul 17 18:33:10.489330 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.489312    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 18:33:10.489330 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.489323    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 18:33:10.490520 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.490509    2563 policy_none.go:47] "Start"
Jul 17 18:33:10.510319 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.510233    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-130.ec2.internal.18c327649f069971  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-130.ec2.internal,UID:ip-10-0-138-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-130.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-130.ec2.internal,},FirstTimestamp:2026-07-17 18:33:10.487071089 +0000 UTC m=+0.431480672,LastTimestamp:2026-07-17 18:33:10.487071089 +0000 UTC m=+0.431480672,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-130.ec2.internal,}"
Jul 17 18:33:10.519597 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.519572    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bz4n2"
Jul 17 18:33:10.522125 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.521996    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-130.ec2.internal.18c327649f06dfd1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-130.ec2.internal,UID:ip-10-0-138-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-130.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-130.ec2.internal,},FirstTimestamp:2026-07-17 18:33:10.487089105 +0000 UTC m=+0.431498688,LastTimestamp:2026-07-17 18:33:10.487089105 +0000 UTC m=+0.431498688,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-130.ec2.internal,}"
Jul 17 18:33:10.528177 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.527777    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-130.ec2.internal.18c327649f0707e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-130.ec2.internal,UID:ip-10-0-138-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-130.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-130.ec2.internal,},FirstTimestamp:2026-07-17 18:33:10.487099365 +0000 UTC m=+0.431508948,LastTimestamp:2026-07-17 18:33:10.487099365 +0000 UTC m=+0.431508948,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-130.ec2.internal,}"
Jul 17 18:33:10.534647 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.534627    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bz4n2"
Jul 17 18:33:10.537147 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.537134    2563 manager.go:337] "Starting Device Plugin manager"
Jul 17 18:33:10.537203 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.537170    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 18:33:10.537203 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.537180    2563 server.go:87] "Starting device plugin registration server"
Jul 17 18:33:10.537476 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.537464    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 18:33:10.537570 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.537476    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 18:33:10.537619 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.537561    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 18:33:10.537665 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.537647    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 18:33:10.537665 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.537657    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 18:33:10.538088 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.538001    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-130.ec2.internal.18c327649f069971\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-130.ec2.internal.18c327649f069971  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-130.ec2.internal,UID:ip-10-0-138-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-130.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-130.ec2.internal,},FirstTimestamp:2026-07-17 18:33:10.487071089 +0000 UTC m=+0.431480672,LastTimestamp:2026-07-17 18:33:10.48879366 +0000 UTC m=+0.433203243,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-130.ec2.internal,}"
Jul 17 18:33:10.538399 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.538379    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 18:33:10.538487 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.538425    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:10.638147 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.638048    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:10.639388 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.639211    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:10.639388 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.639259    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:10.639388 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.639274    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:10.639388 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.639300    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.653415 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.653378    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 18:33:10.654738 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.654717    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 18:33:10.654846 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.654751    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 18:33:10.654846 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.654777    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 18:33:10.654846 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.654825    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 18:33:10.658232 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.658209    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.658330 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.658238    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-130.ec2.internal\": node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:10.675053 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.675031    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:10.713438 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.713400    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:10.755471 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.755437    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal"]
Jul 17 18:33:10.755633 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.755517    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:10.757364 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.757345    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:10.757449 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.757377    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:10.757449 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.757386    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:10.759023 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.759010    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:10.759157 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.759138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.759211 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.759171    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:10.759974 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.759957    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:10.760058 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.759978    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:10.760058 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.759996    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:10.760058 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.760001    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:10.760058 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.760009    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:10.760179 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.760013    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:10.761356 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.761339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.761445 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.761373    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:10.762182 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.762166    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:10.762270 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.762193    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:10.762270 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.762206    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:10.769996 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.769977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dacb0cd9d6d719650fac172fdf23382f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal\" (UID: \"dacb0cd9d6d719650fac172fdf23382f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.770074 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.770003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dacb0cd9d6d719650fac172fdf23382f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal\" (UID: \"dacb0cd9d6d719650fac172fdf23382f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.770074 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.770021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f0ff61e2ec63402244f77cf1bd9480b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-130.ec2.internal\" (UID: \"1f0ff61e2ec63402244f77cf1bd9480b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.789423 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.789404    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-130.ec2.internal\" not found" node="ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.793041 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.793024    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-130.ec2.internal\" not found" node="ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.813943 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.813914    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:10.871027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.870989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f0ff61e2ec63402244f77cf1bd9480b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-130.ec2.internal\" (UID: \"1f0ff61e2ec63402244f77cf1bd9480b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.871027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.871023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dacb0cd9d6d719650fac172fdf23382f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal\" (UID: \"dacb0cd9d6d719650fac172fdf23382f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.871027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.871038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dacb0cd9d6d719650fac172fdf23382f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal\" (UID: \"dacb0cd9d6d719650fac172fdf23382f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.871266 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.871087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dacb0cd9d6d719650fac172fdf23382f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal\" (UID: \"dacb0cd9d6d719650fac172fdf23382f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.871266 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.871093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f0ff61e2ec63402244f77cf1bd9480b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-130.ec2.internal\" (UID: \"1f0ff61e2ec63402244f77cf1bd9480b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.871266 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:10.871132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dacb0cd9d6d719650fac172fdf23382f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal\" (UID: \"dacb0cd9d6d719650fac172fdf23382f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:10.914292 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:10.914222    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:12.048203 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:12.048169    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:12.126329 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.126292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:12.130791 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.130620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:12.149104 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:12.149077    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:12.249699 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:12.249608    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:12.350209 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:12.350169    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:12.437755 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.437722    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 18:33:12.438304 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.437901    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 18:33:12.451067 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:12.451035    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:12.552188 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:12.552094    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-130.ec2.internal\" not found"
Jul 17 18:33:12.574483 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.574440    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 18:28:11 +0000 UTC" deadline="2028-02-02 07:48:44.62797647 +0000 UTC"
Jul 17 18:33:12.574483 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.574474    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13549h15m32.053505798s"
Jul 17 18:33:12.593233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.593208    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:12.603313 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.603284    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:12.617307 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.617289    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 18:33:12.619383 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.619367    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal"
Jul 17 18:33:12.640011 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.639980    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 18:33:12.642856 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.642836    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:12.711062 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:12.711026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacb0cd9d6d719650fac172fdf23382f.slice/crio-61846a55c9d8b32813f72b0c202e0351b38f6ed7e6988fd15cd9f0fd04fb54bd WatchSource:0}: Error finding container 61846a55c9d8b32813f72b0c202e0351b38f6ed7e6988fd15cd9f0fd04fb54bd: Status 404 returned error can't find the container with id 61846a55c9d8b32813f72b0c202e0351b38f6ed7e6988fd15cd9f0fd04fb54bd
Jul 17 18:33:12.711372 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:12.711352    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0ff61e2ec63402244f77cf1bd9480b.slice/crio-e20fe7508c6053033d82ba688e7d73e569cc08d12eb69d5ef870e1e0457d79bd WatchSource:0}: Error finding container e20fe7508c6053033d82ba688e7d73e569cc08d12eb69d5ef870e1e0457d79bd: Status 404 returned error can't find the container with id e20fe7508c6053033d82ba688e7d73e569cc08d12eb69d5ef870e1e0457d79bd
Jul 17 18:33:12.716230 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:12.716217    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:33:13.070393 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.070368    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:13.482334 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.482255    2563 apiserver.go:52] "Watching apiserver"
Jul 17 18:33:13.490466 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.490439    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 18:33:13.491184 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.490825    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx","openshift-network-diagnostics/network-check-target-hclhl","openshift-network-operator/iptables-alerter-mbqt6","openshift-ovn-kubernetes/ovnkube-node-g49xg","kube-system/konnectivity-agent-ltlgb","openshift-cluster-node-tuning-operator/tuned-6rjmc","openshift-dns/node-resolver-dlf6l","openshift-image-registry/node-ca-lsgfn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal","openshift-multus/multus-additional-cni-plugins-mqfcl","openshift-multus/multus-tw7cg","openshift-multus/network-metrics-daemon-hmfvz","kube-system/global-pull-secret-syncer-t88vb","kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal"]
Jul 17 18:33:13.492434 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.492407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.494997 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.494256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.495643 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.495627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.497063 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.497044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.499202 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.499179    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:13.500113 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.500089    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2znwt\""
Jul 17 18:33:13.500366 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.500347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.502318 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.501949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.502489 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.502467    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 18:33:13.503047 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.503027    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 18:33:13.503720 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.503701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:13.503811 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.503771    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:13.504350 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504329    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 18:33:13.504444 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504363    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 18:33:13.504444 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 18:33:13.504444 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504390    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 18:33:13.504689 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504662    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 18:33:13.504784 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 18:33:13.504844 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504822    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-plbvz\""
Jul 17 18:33:13.504899 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 18:33:13.504970 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504952    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:33:13.505833 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.505062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-89dk4\""
Jul 17 18:33:13.505833 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.504963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 18:33:13.505833 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.505225    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 18:33:13.505833 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.505319    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 18:33:13.505833 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.505403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 18:33:13.505833 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.505614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:13.505833 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.505664    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:13.506260 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.506056    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 18:33:13.506260 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.506259    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:33:13.506453 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.506435    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m8m4t\""
Jul 17 18:33:13.506642 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.506626    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 18:33:13.506828 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.506812    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2qql\""
Jul 17 18:33:13.506828 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.506825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dlzlv\""
Jul 17 18:33:13.507600 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.507029    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 18:33:13.507600 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.507063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.507600 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.507160    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 18:33:13.507600 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.507411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 18:33:13.508257 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.508230    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 18:33:13.508543 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.508527    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 18:33:13.509747 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.509166    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 18:33:13.509747 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.509378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k858s\""
Jul 17 18:33:13.509747 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.509646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.510211 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.510194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.510285 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.510262    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:13.512833 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.512799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdcw\" (UniqueName: \"kubernetes.io/projected/7d3069f6-5dc2-40cd-9459-917d4358b97c-kube-api-access-rsdcw\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.512833 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.512829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-hostroot\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.513003 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.512857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.513003 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.512881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-etc-kubernetes\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.513003 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.512907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-cni-bin\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.513003 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.512954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0557c281-a6f2-438f-a8fb-6ff851a4de77-iptables-alerter-script\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.513003 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.512996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-var-lib-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.513281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 18:33:13.513281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73a4619f-1e89-431b-a954-b857d9810083-agent-certs\") pod \"konnectivity-agent-ltlgb\" (UID: \"73a4619f-1e89-431b-a954-b857d9810083\") " pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:13.513281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c764c7-9cd9-40cd-8363-87505d4d1df3-host\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.513281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qs6\" (UniqueName: \"kubernetes.io/projected/e0553119-5c66-4b8e-9e4b-c68c8f06af08-kube-api-access-k2qs6\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:13.513281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-run\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.513281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b95b0065-a98f-4381-8d0f-eff4aef03eb7-cni-binary-copy\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.513281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-run-ovn-kubernetes\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.513281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-kubernetes\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysctl-d\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3069f6-5dc2-40cd-9459-917d4358b97c-tmp-dir\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-slash\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-host\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-env-overrides\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8vg8\" (UniqueName: \"kubernetes.io/projected/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-kube-api-access-n8vg8\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-modprobe-d\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-lib-modules\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-multus-certs\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4vw\" (UniqueName: \"kubernetes.io/projected/b95b0065-a98f-4381-8d0f-eff4aef03eb7-kube-api-access-2x4vw\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-systemd-units\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-k8s-cni-cncf-io\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.513650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-kubelet\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-daemon-config\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-cni-netd\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-netns\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-conf-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0557c281-a6f2-438f-a8fb-6ff851a4de77-host-slash\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-node-log\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovnkube-config\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73a4619f-1e89-431b-a954-b857d9810083-konnectivity-ca\") pod \"konnectivity-agent-ltlgb\" (UID: \"73a4619f-1e89-431b-a954-b857d9810083\") " pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-cni-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354b6941-9665-4e1c-8115-6e3d01a7dc28-tmp\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k926w\" (UniqueName: \"kubernetes.io/projected/354b6941-9665-4e1c-8115-6e3d01a7dc28-kube-api-access-k926w\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-etc-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.513990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-systemd\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-cni-multus\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-run-netns\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.514423 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-systemd\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37c764c7-9cd9-40cd-8363-87505d4d1df3-serviceca\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwj96\" (UniqueName: \"kubernetes.io/projected/0557c281-a6f2-438f-a8fb-6ff851a4de77-kube-api-access-bwj96\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-log-socket\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysctl-conf\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-tuned\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d3069f6-5dc2-40cd-9459-917d4358b97c-hosts-file\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c4mp\" (UniqueName: \"kubernetes.io/projected/37c764c7-9cd9-40cd-8363-87505d4d1df3-kube-api-access-4c4mp\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-socket-dir-parent\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-ovn\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovn-node-metrics-cert\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-cnibin\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-sys\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-var-lib-kubelet\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-system-cni-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovnkube-script-lib\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysconfig\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.515035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-os-release\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.515825 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-cni-bin\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.515825 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.514341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-kubelet\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.515825 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.515047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 18:33:13.515825 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.515357    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mhxkn\""
Jul 17 18:33:13.515825 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.515547    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 18:33:13.516205 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.516184    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 18:33:13.516568 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.516545    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxr6b\""
Jul 17 18:33:13.516893 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.516873    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 18:33:13.534845 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.534817    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 18:33:13.604752 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.604684    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 18:33:13.607615 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.607591    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vl96m"
Jul 17 18:33:13.614545 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k926w\" (UniqueName: \"kubernetes.io/projected/354b6941-9665-4e1c-8115-6e3d01a7dc28-kube-api-access-k926w\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.614662 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-etc-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.614662 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmff\" (UniqueName: \"kubernetes.io/projected/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-kube-api-access-rhmff\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.614662 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-systemd\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.614662 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-cni-multus\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.614662 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-run-netns\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.614662 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-systemd\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.614662 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37c764c7-9cd9-40cd-8363-87505d4d1df3-serviceca\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwj96\" (UniqueName: \"kubernetes.io/projected/0557c281-a6f2-438f-a8fb-6ff851a4de77-kube-api-access-bwj96\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-log-socket\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysctl-conf\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-tuned\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d3069f6-5dc2-40cd-9459-917d4358b97c-hosts-file\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4mp\" (UniqueName: \"kubernetes.io/projected/37c764c7-9cd9-40cd-8363-87505d4d1df3-kube-api-access-4c4mp\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-socket-dir-parent\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-ovn\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovn-node-metrics-cert\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-cnibin\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-sys\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-var-lib-kubelet\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-system-cni-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovnkube-script-lib\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.614945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysconfig\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-os-release\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-cni-bin\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.614998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-kubelet\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0baa97c-7dca-43b5-b52d-680c32593d70-dbus\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdcw\" (UniqueName: \"kubernetes.io/projected/7d3069f6-5dc2-40cd-9459-917d4358b97c-kube-api-access-rsdcw\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-hostroot\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-sys-fs\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-etc-kubernetes\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-cni-bin\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-system-cni-dir\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0557c281-a6f2-438f-a8fb-6ff851a4de77-iptables-alerter-script\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-var-lib-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73a4619f-1e89-431b-a954-b857d9810083-agent-certs\") pod \"konnectivity-agent-ltlgb\" (UID: \"73a4619f-1e89-431b-a954-b857d9810083\") " pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-device-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.615461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0baa97c-7dca-43b5-b52d-680c32593d70-kubelet-config\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c764c7-9cd9-40cd-8363-87505d4d1df3-host\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qs6\" (UniqueName: \"kubernetes.io/projected/e0553119-5c66-4b8e-9e4b-c68c8f06af08-kube-api-access-k2qs6\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-socket-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-run\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b95b0065-a98f-4381-8d0f-eff4aef03eb7-cni-binary-copy\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-run-ovn-kubernetes\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-kubernetes\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysctl-d\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3069f6-5dc2-40cd-9459-917d4358b97c-tmp-dir\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-slash\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4244\" (UniqueName: \"kubernetes.io/projected/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-kube-api-access-j4244\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-host\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:13.615923 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-env-overrides\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8vg8\" (UniqueName: \"kubernetes.io/projected/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-kube-api-access-n8vg8\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-os-release\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-registration-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-modprobe-d\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-lib-modules\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-multus-certs\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4vw\" (UniqueName: \"kubernetes.io/projected/b95b0065-a98f-4381-8d0f-eff4aef03eb7-kube-api-access-2x4vw\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-systemd-units\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cnibin\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-k8s-cni-cncf-io\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-kubelet\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-daemon-config\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-cni-netd\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:13.616539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-netns\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-conf-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0557c281-a6f2-438f-a8fb-6ff851a4de77-host-slash\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-node-log\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovnkube-config\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73a4619f-1e89-431b-a954-b857d9810083-konnectivity-ca\") pod \"konnectivity-agent-ltlgb\" (UID: \"73a4619f-1e89-431b-a954-b857d9810083\") " pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-cni-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.615981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354b6941-9665-4e1c-8115-6e3d01a7dc28-tmp\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-kubernetes\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-modprobe-d\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c764c7-9cd9-40cd-8363-87505d4d1df3-host\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616349    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-etc-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3069f6-5dc2-40cd-9459-917d4358b97c-tmp-dir\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-systemd\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-slash\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-cni-multus\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.617357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-host\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.616927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-log-socket\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysctl-conf\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37c764c7-9cd9-40cd-8363-87505d4d1df3-serviceca\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-env-overrides\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b95b0065-a98f-4381-8d0f-eff4aef03eb7-cni-binary-copy\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-os-release\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysconfig\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-run-ovn-kubernetes\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-kubelet\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-run\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-sysctl-d\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-systemd\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.617601    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-cni-bin\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-run-netns\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.617673    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:14.117652475 +0000 UTC m=+3.028479233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-cni-netd\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.618248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-hostroot\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-etc-kubernetes\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-lib-modules\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-cni-bin\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-k8s-cni-cncf-io\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-multus-certs\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.617922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-var-lib-kubelet\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-systemd-units\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-var-lib-openvswitch\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0557c281-a6f2-438f-a8fb-6ff851a4de77-host-slash\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-host-run-netns\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-conf-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0557c281-a6f2-438f-a8fb-6ff851a4de77-iptables-alerter-script\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d3069f6-5dc2-40cd-9459-917d4358b97c-hosts-file\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73a4619f-1e89-431b-a954-b857d9810083-konnectivity-ca\") pod \"konnectivity-agent-ltlgb\" (UID: \"73a4619f-1e89-431b-a954-b857d9810083\") " pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-socket-dir-parent\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.619109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-node-log\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.620020 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.618994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-run-ovn\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.620020 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.619407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovnkube-config\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.620020 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.619544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-cni-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.620020 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.619597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-var-lib-kubelet\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.620020 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.619646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-cnibin\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.620020 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.619693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/354b6941-9665-4e1c-8115-6e3d01a7dc28-sys\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.620020 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.619751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b95b0065-a98f-4381-8d0f-eff4aef03eb7-system-cni-dir\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.620381 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.620238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovnkube-script-lib\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.620381 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.620303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354b6941-9665-4e1c-8115-6e3d01a7dc28-tmp\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.620587 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.620559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b95b0065-a98f-4381-8d0f-eff4aef03eb7-multus-daemon-config\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.621008 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.620986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73a4619f-1e89-431b-a954-b857d9810083-agent-certs\") pod \"konnectivity-agent-ltlgb\" (UID: \"73a4619f-1e89-431b-a954-b857d9810083\") " pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:13.621954 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.621558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-ovn-node-metrics-cert\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.622066 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.622042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/354b6941-9665-4e1c-8115-6e3d01a7dc28-etc-tuned\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.639586 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.639557    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vl96m"
Jul 17 18:33:13.642533 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.642504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k926w\" (UniqueName: \"kubernetes.io/projected/354b6941-9665-4e1c-8115-6e3d01a7dc28-kube-api-access-k926w\") pod \"tuned-6rjmc\" (UID: \"354b6941-9665-4e1c-8115-6e3d01a7dc28\") " pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.650011 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.649954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwj96\" (UniqueName: \"kubernetes.io/projected/0557c281-a6f2-438f-a8fb-6ff851a4de77-kube-api-access-bwj96\") pod \"iptables-alerter-mbqt6\" (UID: \"0557c281-a6f2-438f-a8fb-6ff851a4de77\") " pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.650143 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.650104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdcw\" (UniqueName: \"kubernetes.io/projected/7d3069f6-5dc2-40cd-9459-917d4358b97c-kube-api-access-rsdcw\") pod \"node-resolver-dlf6l\" (UID: \"7d3069f6-5dc2-40cd-9459-917d4358b97c\") " pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.652521 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.652496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qs6\" (UniqueName: \"kubernetes.io/projected/e0553119-5c66-4b8e-9e4b-c68c8f06af08-kube-api-access-k2qs6\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:13.652794 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.652772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4vw\" (UniqueName: \"kubernetes.io/projected/b95b0065-a98f-4381-8d0f-eff4aef03eb7-kube-api-access-2x4vw\") pod \"multus-tw7cg\" (UID: \"b95b0065-a98f-4381-8d0f-eff4aef03eb7\") " pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.654506 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.654464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4mp\" (UniqueName: \"kubernetes.io/projected/37c764c7-9cd9-40cd-8363-87505d4d1df3-kube-api-access-4c4mp\") pod \"node-ca-lsgfn\" (UID: \"37c764c7-9cd9-40cd-8363-87505d4d1df3\") " pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.656253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.656229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8vg8\" (UniqueName: \"kubernetes.io/projected/eb5d4daf-45c2-42f2-92fc-21c7e1e30083-kube-api-access-n8vg8\") pod \"ovnkube-node-g49xg\" (UID: \"eb5d4daf-45c2-42f2-92fc-21c7e1e30083\") " pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.694095 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.694039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal" event={"ID":"dacb0cd9d6d719650fac172fdf23382f","Type":"ContainerStarted","Data":"61846a55c9d8b32813f72b0c202e0351b38f6ed7e6988fd15cd9f0fd04fb54bd"}
Jul 17 18:33:13.695287 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.695258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal" event={"ID":"1f0ff61e2ec63402244f77cf1bd9480b","Type":"ContainerStarted","Data":"e20fe7508c6053033d82ba688e7d73e569cc08d12eb69d5ef870e1e0457d79bd"}
Jul 17 18:33:13.704928 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.704907    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:13.716841 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.716807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.716992 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.716858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0baa97c-7dca-43b5-b52d-680c32593d70-dbus\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.716992 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.716888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.716992 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.716909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-sys-fs\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.716992 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.716937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-system-cni-dir\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.716992 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.716961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-device-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.716992 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.716981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0baa97c-7dca-43b5-b52d-680c32593d70-kubelet-config\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-sys-fs\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-socket-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-system-cni-dir\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-device-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0baa97c-7dca-43b5-b52d-680c32593d70-dbus\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.717184    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.717238    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret podName:f0baa97c-7dca-43b5-b52d-680c32593d70 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:14.217218809 +0000 UTC m=+3.128045550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret") pod "global-pull-secret-syncer-t88vb" (UID: "f0baa97c-7dca-43b5-b52d-680c32593d70") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0baa97c-7dca-43b5-b52d-680c32593d70-kubelet-config\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4244\" (UniqueName: \"kubernetes.io/projected/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-kube-api-access-j4244\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-socket-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-os-release\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-registration-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cnibin\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-os-release\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cnibin\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmff\" (UniqueName: \"kubernetes.io/projected/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-kube-api-access-rhmff\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-registration-dir\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.717950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.717880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.744606 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.744519    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:13.744606 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.744555    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:13.744606 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.744569    2563 projected.go:196] Error preparing data for projected volume kube-api-access-2st2z for pod openshift-network-diagnostics/network-check-target-hclhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:13.744854 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:13.744647    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z podName:2737d09a-42b8-481d-8fae-378b80ca56ca nodeName:}" failed. No retries permitted until 2026-07-17 18:33:14.244627689 +0000 UTC m=+3.155454447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2st2z" (UniqueName: "kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z") pod "network-check-target-hclhl" (UID: "2737d09a-42b8-481d-8fae-378b80ca56ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:13.747265 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.747240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4244\" (UniqueName: \"kubernetes.io/projected/564a83b7-e7d0-48fc-b64c-ad02f0b688a8-kube-api-access-j4244\") pod \"multus-additional-cni-plugins-mqfcl\" (UID: \"564a83b7-e7d0-48fc-b64c-ad02f0b688a8\") " pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.748661 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.748633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmff\" (UniqueName: \"kubernetes.io/projected/96fccbcb-c7f6-4dd9-acb4-213de051fb7e-kube-api-access-rhmff\") pod \"aws-ebs-csi-driver-node-dp9sx\" (UID: \"96fccbcb-c7f6-4dd9-acb4-213de051fb7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:13.806817 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.806786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dlf6l"
Jul 17 18:33:13.814108 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.814075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tw7cg"
Jul 17 18:33:13.826107 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.826078    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mbqt6"
Jul 17 18:33:13.835998 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.835977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:13.842849 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.842822    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:13.850262 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.850236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6rjmc"
Jul 17 18:33:13.856215 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.856194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lsgfn"
Jul 17 18:33:13.862086 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.862064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mqfcl"
Jul 17 18:33:13.868287 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:13.868260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx"
Jul 17 18:33:14.120230 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.120147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:14.120400 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.120278    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:14.120400 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.120339    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:15.120324559 +0000 UTC m=+4.031151300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:14.221222 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.221189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:14.221403 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.221378    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:14.221483 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.221472    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret podName:f0baa97c-7dca-43b5-b52d-680c32593d70 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:15.221450861 +0000 UTC m=+4.132277619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret") pod "global-pull-secret-syncer-t88vb" (UID: "f0baa97c-7dca-43b5-b52d-680c32593d70") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:14.322478 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.322438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:14.322623 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.322594    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:14.322623 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.322615    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:14.322623 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.322624    2563 projected.go:196] Error preparing data for projected volume kube-api-access-2st2z for pod openshift-network-diagnostics/network-check-target-hclhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:14.322759 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.322684    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z podName:2737d09a-42b8-481d-8fae-378b80ca56ca nodeName:}" failed. No retries permitted until 2026-07-17 18:33:15.322668213 +0000 UTC m=+4.233494974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2st2z" (UniqueName: "kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z") pod "network-check-target-hclhl" (UID: "2737d09a-42b8-481d-8fae-378b80ca56ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:14.569959 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.569928    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3069f6_5dc2_40cd_9459_917d4358b97c.slice/crio-de1b7c6b334f6f1e1e1e654ac7b7f8a0510ebc2582c4818e88a5921a5976889f WatchSource:0}: Error finding container de1b7c6b334f6f1e1e1e654ac7b7f8a0510ebc2582c4818e88a5921a5976889f: Status 404 returned error can't find the container with id de1b7c6b334f6f1e1e1e654ac7b7f8a0510ebc2582c4818e88a5921a5976889f
Jul 17 18:33:14.570792 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.570764    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5d4daf_45c2_42f2_92fc_21c7e1e30083.slice/crio-30bbba7420377dbfdae2df51ed332afe07be39bce8df76aaebb7b08182cc99fe WatchSource:0}: Error finding container 30bbba7420377dbfdae2df51ed332afe07be39bce8df76aaebb7b08182cc99fe: Status 404 returned error can't find the container with id 30bbba7420377dbfdae2df51ed332afe07be39bce8df76aaebb7b08182cc99fe
Jul 17 18:33:14.573797 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.573777    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95b0065_a98f_4381_8d0f_eff4aef03eb7.slice/crio-3c612ddeb7c5fbaac7001ea786be9f0b9a7250c441294defb8fada3b474b0742 WatchSource:0}: Error finding container 3c612ddeb7c5fbaac7001ea786be9f0b9a7250c441294defb8fada3b474b0742: Status 404 returned error can't find the container with id 3c612ddeb7c5fbaac7001ea786be9f0b9a7250c441294defb8fada3b474b0742
Jul 17 18:33:14.576043 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.576016    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564a83b7_e7d0_48fc_b64c_ad02f0b688a8.slice/crio-f0904f3c31c51e79dd4983ad2072c0083829fdf1e7714200131a833bd69e618b WatchSource:0}: Error finding container f0904f3c31c51e79dd4983ad2072c0083829fdf1e7714200131a833bd69e618b: Status 404 returned error can't find the container with id f0904f3c31c51e79dd4983ad2072c0083829fdf1e7714200131a833bd69e618b
Jul 17 18:33:14.586932 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.586911    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354b6941_9665_4e1c_8115_6e3d01a7dc28.slice/crio-34bb9c709d4021acabf265d55c29cc46481be79f341e67c79e21e16f217a62b7 WatchSource:0}: Error finding container 34bb9c709d4021acabf265d55c29cc46481be79f341e67c79e21e16f217a62b7: Status 404 returned error can't find the container with id 34bb9c709d4021acabf265d55c29cc46481be79f341e67c79e21e16f217a62b7
Jul 17 18:33:14.589822 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.589787    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c764c7_9cd9_40cd_8363_87505d4d1df3.slice/crio-f1e9af84fc7feb86e81170e1f10b2dafe9537859e233819db1af4a2e47c68e8a WatchSource:0}: Error finding container f1e9af84fc7feb86e81170e1f10b2dafe9537859e233819db1af4a2e47c68e8a: Status 404 returned error can't find the container with id f1e9af84fc7feb86e81170e1f10b2dafe9537859e233819db1af4a2e47c68e8a
Jul 17 18:33:14.590618 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.590580    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0557c281_a6f2_438f_a8fb_6ff851a4de77.slice/crio-3f9270f7cde355518ded3627109de59940499db447bb707cee94360d2c98746e WatchSource:0}: Error finding container 3f9270f7cde355518ded3627109de59940499db447bb707cee94360d2c98746e: Status 404 returned error can't find the container with id 3f9270f7cde355518ded3627109de59940499db447bb707cee94360d2c98746e
Jul 17 18:33:14.592509 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.592484    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a4619f_1e89_431b_a954_b857d9810083.slice/crio-36866f3ade3aeedf1e96a43be1e88fbf698aede4076d843d79c5da8c6b40f4fa WatchSource:0}: Error finding container 36866f3ade3aeedf1e96a43be1e88fbf698aede4076d843d79c5da8c6b40f4fa: Status 404 returned error can't find the container with id 36866f3ade3aeedf1e96a43be1e88fbf698aede4076d843d79c5da8c6b40f4fa
Jul 17 18:33:14.596064 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:14.596042    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fccbcb_c7f6_4dd9_acb4_213de051fb7e.slice/crio-ad0158168b20db968c62d9f58e3d6e04a692e66413ac7d9e915d65f7c5fad597 WatchSource:0}: Error finding container ad0158168b20db968c62d9f58e3d6e04a692e66413ac7d9e915d65f7c5fad597: Status 404 returned error can't find the container with id ad0158168b20db968c62d9f58e3d6e04a692e66413ac7d9e915d65f7c5fad597
Jul 17 18:33:14.642930 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.642899    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:28:13 +0000 UTC" deadline="2028-01-22 05:44:53.065833688 +0000 UTC"
Jul 17 18:33:14.642930 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.642924    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13283h11m38.422912266s"
Jul 17 18:33:14.688738 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.688703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:14.688853 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.688806    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:14.688899 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.688862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:14.688959 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:14.688918    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:14.701610 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.701585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dlf6l" event={"ID":"7d3069f6-5dc2-40cd-9459-917d4358b97c","Type":"ContainerStarted","Data":"de1b7c6b334f6f1e1e1e654ac7b7f8a0510ebc2582c4818e88a5921a5976889f"}
Jul 17 18:33:14.707496 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.707468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6rjmc" event={"ID":"354b6941-9665-4e1c-8115-6e3d01a7dc28","Type":"ContainerStarted","Data":"34bb9c709d4021acabf265d55c29cc46481be79f341e67c79e21e16f217a62b7"}
Jul 17 18:33:14.708466 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.708443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lsgfn" event={"ID":"37c764c7-9cd9-40cd-8363-87505d4d1df3","Type":"ContainerStarted","Data":"f1e9af84fc7feb86e81170e1f10b2dafe9537859e233819db1af4a2e47c68e8a"}
Jul 17 18:33:14.714035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.714010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerStarted","Data":"f0904f3c31c51e79dd4983ad2072c0083829fdf1e7714200131a833bd69e618b"}
Jul 17 18:33:14.715992 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.715972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tw7cg" event={"ID":"b95b0065-a98f-4381-8d0f-eff4aef03eb7","Type":"ContainerStarted","Data":"3c612ddeb7c5fbaac7001ea786be9f0b9a7250c441294defb8fada3b474b0742"}
Jul 17 18:33:14.717523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.717504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"30bbba7420377dbfdae2df51ed332afe07be39bce8df76aaebb7b08182cc99fe"}
Jul 17 18:33:14.719658 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.719635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal" event={"ID":"1f0ff61e2ec63402244f77cf1bd9480b","Type":"ContainerStarted","Data":"b25d4dcaedb86d9892842fb7a8fc0c4753d1f5ce40bd94f93565dedbc64a77ba"}
Jul 17 18:33:14.721314 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.721296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx" event={"ID":"96fccbcb-c7f6-4dd9-acb4-213de051fb7e","Type":"ContainerStarted","Data":"ad0158168b20db968c62d9f58e3d6e04a692e66413ac7d9e915d65f7c5fad597"}
Jul 17 18:33:14.722435 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.722418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ltlgb" event={"ID":"73a4619f-1e89-431b-a954-b857d9810083","Type":"ContainerStarted","Data":"36866f3ade3aeedf1e96a43be1e88fbf698aede4076d843d79c5da8c6b40f4fa"}
Jul 17 18:33:14.723344 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.723324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mbqt6" event={"ID":"0557c281-a6f2-438f-a8fb-6ff851a4de77","Type":"ContainerStarted","Data":"3f9270f7cde355518ded3627109de59940499db447bb707cee94360d2c98746e"}
Jul 17 18:33:14.758916 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:14.758863    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-130.ec2.internal" podStartSLOduration=2.75884668 podStartE2EDuration="2.75884668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:14.758586034 +0000 UTC m=+3.669412794" watchObservedRunningTime="2026-07-17 18:33:14.75884668 +0000 UTC m=+3.669673440"
Jul 17 18:33:15.127894 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:15.127857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:15.128084 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.128068    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:15.128191 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.128149    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:17.128111482 +0000 UTC m=+6.038938246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:15.229136 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:15.229089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:15.229377 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.229320    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:15.229452 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.229382    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret podName:f0baa97c-7dca-43b5-b52d-680c32593d70 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:17.229364723 +0000 UTC m=+6.140191476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret") pod "global-pull-secret-syncer-t88vb" (UID: "f0baa97c-7dca-43b5-b52d-680c32593d70") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:15.329574 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:15.329534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:15.329772 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.329727    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:15.329772 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.329746    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:15.329772 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.329759    2563 projected.go:196] Error preparing data for projected volume kube-api-access-2st2z for pod openshift-network-diagnostics/network-check-target-hclhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:15.329925 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.329818    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z podName:2737d09a-42b8-481d-8fae-378b80ca56ca nodeName:}" failed. No retries permitted until 2026-07-17 18:33:17.329799491 +0000 UTC m=+6.240626246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2st2z" (UniqueName: "kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z") pod "network-check-target-hclhl" (UID: "2737d09a-42b8-481d-8fae-378b80ca56ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:15.643657 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:15.643592    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:28:13 +0000 UTC" deadline="2028-01-04 01:52:49.097290297 +0000 UTC"
Jul 17 18:33:15.643657 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:15.643654    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12847h19m33.453640361s"
Jul 17 18:33:15.691268 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:15.691235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:15.691440 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:15.691378    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:15.752866 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:15.752809    2563 generic.go:356] "Generic (PLEG): container finished" podID="dacb0cd9d6d719650fac172fdf23382f" containerID="ed4b2e6bac190fc92278eb5327a78742e214d328a99e2b594f7aa96b3fed1983" exitCode=0
Jul 17 18:33:15.753834 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:15.753802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal" event={"ID":"dacb0cd9d6d719650fac172fdf23382f","Type":"ContainerDied","Data":"ed4b2e6bac190fc92278eb5327a78742e214d328a99e2b594f7aa96b3fed1983"}
Jul 17 18:33:16.688622 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:16.688588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:16.689076 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:16.688730    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:16.689187 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:16.689167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:16.689306 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:16.689284    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:16.787742 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:16.787692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal" event={"ID":"dacb0cd9d6d719650fac172fdf23382f","Type":"ContainerStarted","Data":"1bedc57698b6079d328d94d96d25efe51e82f85c52bf812aecab77b3968db85a"}
Jul 17 18:33:17.146200 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:17.145549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:17.146200 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.145701    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:17.146200 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.145764    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:21.145745814 +0000 UTC m=+10.056572566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:17.246707 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:17.246666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:17.246892 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.246846    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:17.246950 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.246911    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret podName:f0baa97c-7dca-43b5-b52d-680c32593d70 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:21.246891425 +0000 UTC m=+10.157718180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret") pod "global-pull-secret-syncer-t88vb" (UID: "f0baa97c-7dca-43b5-b52d-680c32593d70") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:17.347250 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:17.347215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:17.347431 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.347389    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:17.347431 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.347414    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:17.347431 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.347426    2563 projected.go:196] Error preparing data for projected volume kube-api-access-2st2z for pod openshift-network-diagnostics/network-check-target-hclhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:17.347591 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.347479    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z podName:2737d09a-42b8-481d-8fae-378b80ca56ca nodeName:}" failed. No retries permitted until 2026-07-17 18:33:21.347462183 +0000 UTC m=+10.258288925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2st2z" (UniqueName: "kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z") pod "network-check-target-hclhl" (UID: "2737d09a-42b8-481d-8fae-378b80ca56ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:17.691613 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:17.691578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:17.692070 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:17.691743    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:18.689234 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:18.688511    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:18.689234 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:18.688627    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:18.689234 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:18.688744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:18.689234 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:18.688848    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:19.688937 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:19.688850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:19.689323 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:19.688990    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:20.688703 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:20.688665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:20.688899 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:20.688677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:20.688899 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:20.688788    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:20.688899 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:20.688868    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:21.182223 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:21.181653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:21.182223 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.181808    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:21.182223 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.181875    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:29.181854638 +0000 UTC m=+18.092681383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:21.282256 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:21.282222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:21.282443 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.282402    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:21.282503 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.282466    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret podName:f0baa97c-7dca-43b5-b52d-680c32593d70 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:29.282449538 +0000 UTC m=+18.193276276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret") pod "global-pull-secret-syncer-t88vb" (UID: "f0baa97c-7dca-43b5-b52d-680c32593d70") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:21.383644 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:21.383608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:21.383829 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.383771    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:21.383829 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.383787    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:21.383829 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.383798    2563 projected.go:196] Error preparing data for projected volume kube-api-access-2st2z for pod openshift-network-diagnostics/network-check-target-hclhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:21.383987 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.383846    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z podName:2737d09a-42b8-481d-8fae-378b80ca56ca nodeName:}" failed. No retries permitted until 2026-07-17 18:33:29.383830454 +0000 UTC m=+18.294657193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2st2z" (UniqueName: "kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z") pod "network-check-target-hclhl" (UID: "2737d09a-42b8-481d-8fae-378b80ca56ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:21.694151 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:21.690302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:21.694151 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:21.690403    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:22.689613 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:22.689574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:22.689790 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:22.689697    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:22.690087 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:22.690068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:22.690213 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:22.690171    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:23.689539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:23.689504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:23.689983 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:23.689688    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:24.689164 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:24.689131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:24.689324 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:24.689140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:24.689324 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:24.689260    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:24.689433 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:24.689350    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:25.694179 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:25.694147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:25.694617 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:25.694265    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:26.689582 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:26.689547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:26.689727 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:26.689552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:26.689800 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:26.689763    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:26.689800 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:26.689651    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:27.689037 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:27.689000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:27.689555 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:27.689167    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:28.688993 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:28.688953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:28.688993 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:28.688987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:28.689460 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:28.689108    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:28.689460 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:28.689111    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:29.248450 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:29.248412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:29.248666 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.248533    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:29.248666 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.248593    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:45.248576619 +0000 UTC m=+34.159403366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:29.349499 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:29.349453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:29.349671 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.349609    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:29.349741 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.349694    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret podName:f0baa97c-7dca-43b5-b52d-680c32593d70 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:45.349671339 +0000 UTC m=+34.260498081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret") pod "global-pull-secret-syncer-t88vb" (UID: "f0baa97c-7dca-43b5-b52d-680c32593d70") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:29.450326 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:29.450286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:29.450488 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.450436    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:29.450488 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.450467    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:29.450488 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.450482    2563 projected.go:196] Error preparing data for projected volume kube-api-access-2st2z for pod openshift-network-diagnostics/network-check-target-hclhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:29.450642 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.450550    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z podName:2737d09a-42b8-481d-8fae-378b80ca56ca nodeName:}" failed. No retries permitted until 2026-07-17 18:33:45.450530723 +0000 UTC m=+34.361357474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2st2z" (UniqueName: "kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z") pod "network-check-target-hclhl" (UID: "2737d09a-42b8-481d-8fae-378b80ca56ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:29.688894 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:29.688863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:29.689074 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:29.689007    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:30.688640 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:30.688597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:30.689077 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:30.688596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:30.689077 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:30.688738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:30.689077 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:30.688834    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:31.690551 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:31.690510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:31.690963 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:31.690644    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:32.689407 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:32.689368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:32.689407 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:32.689403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:32.689659 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:32.689516    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:32.689659 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:32.689630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:33.689451 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.689216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:33.689944 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:33.689571    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:33.823848 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.823822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6rjmc" event={"ID":"354b6941-9665-4e1c-8115-6e3d01a7dc28","Type":"ContainerStarted","Data":"a6ff1094c6397c91a5a006397ddb79441e2c0a614a3ad060b27e4c5eb1cc3b57"}
Jul 17 18:33:33.825289 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.825260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lsgfn" event={"ID":"37c764c7-9cd9-40cd-8363-87505d4d1df3","Type":"ContainerStarted","Data":"b2ac46c942804a0ced7225717f2831fa9975d18ccd7ecfc7759cac29521d4cd2"}
Jul 17 18:33:33.826560 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.826536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerStarted","Data":"4f4b63d28ab864ee568984b480625f57480feab87289205a5e526f4af3dc4a4f"}
Jul 17 18:33:33.827988 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.827961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tw7cg" event={"ID":"b95b0065-a98f-4381-8d0f-eff4aef03eb7","Type":"ContainerStarted","Data":"ec7bdcf71326685494474313f55a817c3918e95448efbc07a82909ac570e134b"}
Jul 17 18:33:33.829999 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.829978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"9da2a91696835731f70639f0f06036b6c10be704d593009afc8ef04eb8f3c146"}
Jul 17 18:33:33.830103 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.830049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"a54bfce9125b65bc8cf0ef4b5b51ff79a23e76a9732db1883e1dda08e717de94"}
Jul 17 18:33:33.831255 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.831235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx" event={"ID":"96fccbcb-c7f6-4dd9-acb4-213de051fb7e","Type":"ContainerStarted","Data":"ec7cca1401426c544145511dd83565d8760e7bcf5b1b80254a24d504e2f33d67"}
Jul 17 18:33:33.832827 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.832803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ltlgb" event={"ID":"73a4619f-1e89-431b-a954-b857d9810083","Type":"ContainerStarted","Data":"e130a17984fc94f790544ad9e08c4c67211c549eb669f9686b3c8ec0cfce9b4d"}
Jul 17 18:33:33.834206 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.834187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dlf6l" event={"ID":"7d3069f6-5dc2-40cd-9459-917d4358b97c","Type":"ContainerStarted","Data":"4171c95cf99e0f16d866e6ece1ce82f2ff9e9c24d63580b4cb5b64483e380b8d"}
Jul 17 18:33:33.850739 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.850691    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6rjmc" podStartSLOduration=4.009725496 podStartE2EDuration="22.85067849s" totalImagesPullingTime="18.840952994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:33.850215063 +0000 UTC m=+22.761041843" watchObservedRunningTime="2026-07-17 18:33:33.85067849 +0000 UTC m=+22.761505251"
Jul 17 18:33:33.850860 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.850841    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-130.ec2.internal" podStartSLOduration=21.85083466 podStartE2EDuration="21.85083466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:16.830898009 +0000 UTC m=+5.741724770" watchObservedRunningTime="2026-07-17 18:33:33.85083466 +0000 UTC m=+22.761661421"
Jul 17 18:33:33.869665 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.869617    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lsgfn" podStartSLOduration=4.032771769 podStartE2EDuration="22.869601844s" totalImagesPullingTime="18.836830075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:33.869203442 +0000 UTC m=+22.780030224" watchObservedRunningTime="2026-07-17 18:33:33.869601844 +0000 UTC m=+22.780428606"
Jul 17 18:33:33.958594 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:33.958536    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dlf6l" podStartSLOduration=4.102113203 podStartE2EDuration="22.958513405s" totalImagesPullingTime="18.856400202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:33.93059612 +0000 UTC m=+22.841422891" watchObservedRunningTime="2026-07-17 18:33:33.958513405 +0000 UTC m=+22.869340168"
Jul 17 18:33:34.001660 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.001607    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tw7cg" podStartSLOduration=4.111717959 podStartE2EDuration="23.001587905s" totalImagesPullingTime="18.889869946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:33.962114263 +0000 UTC m=+22.872955538" watchObservedRunningTime="2026-07-17 18:33:34.001587905 +0000 UTC m=+22.912414644"
Jul 17 18:33:34.002066 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.002042    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ltlgb" podStartSLOduration=8.293859905 podStartE2EDuration="23.002035348s" totalImagesPullingTime="14.708175443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:34.001859296 +0000 UTC m=+22.912686056" watchObservedRunningTime="2026-07-17 18:33:34.002035348 +0000 UTC m=+22.912862149"
Jul 17 18:33:34.689476 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.689279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:34.690059 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.689333    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:34.690059 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:34.689549    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:34.690059 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:34.689647    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:34.838258 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.838226    2563 generic.go:356] "Generic (PLEG): container finished" podID="564a83b7-e7d0-48fc-b64c-ad02f0b688a8" containerID="4f4b63d28ab864ee568984b480625f57480feab87289205a5e526f4af3dc4a4f" exitCode=0
Jul 17 18:33:34.838418 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.838261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerDied","Data":"4f4b63d28ab864ee568984b480625f57480feab87289205a5e526f4af3dc4a4f"}
Jul 17 18:33:34.840755 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.840739    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:33:34.841079 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.841062    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb5d4daf-45c2-42f2-92fc-21c7e1e30083" containerID="9da2a91696835731f70639f0f06036b6c10be704d593009afc8ef04eb8f3c146" exitCode=1
Jul 17 18:33:34.841194 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.841171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerDied","Data":"9da2a91696835731f70639f0f06036b6c10be704d593009afc8ef04eb8f3c146"}
Jul 17 18:33:34.841277 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.841261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"31cd4a6b9991bc8096d16de490057433031ddf7170b03220d11afb1f75354ff8"}
Jul 17 18:33:34.841330 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.841319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"4440645fa26d6b117b35dd7be2975413b55d4d3f964237ff38fe69b1e30a09a9"}
Jul 17 18:33:34.841380 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.841368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"fbffa5fcbe79672478d1627cf339c773409613e226d687bdb0a3e41f7e94288c"}
Jul 17 18:33:34.841429 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:34.841417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"81659cc159385ff241fed59445c46c445a9d4d9c2a97406b2e22d279de031b41"}
Jul 17 18:33:35.119851 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.119826    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 18:33:35.154193 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.154157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:35.154777 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.154758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:35.583198 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.583069    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T18:33:35.119845661Z","UUID":"eb5f1ca7-93dc-402a-a80c-9d43c9fecce2","Handler":null,"Name":"","Endpoint":""}
Jul 17 18:33:35.584865 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.584839    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 18:33:35.585001 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.584884    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 18:33:35.689792 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.689402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:35.689792 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:35.689526    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:35.846386 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.846348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx" event={"ID":"96fccbcb-c7f6-4dd9-acb4-213de051fb7e","Type":"ContainerStarted","Data":"e0531fd0fdf6d8f9711aefd6f1bbc7962751bf1dbe443acd73cb2f368a7d9ef6"}
Jul 17 18:33:35.850074 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.850041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mbqt6" event={"ID":"0557c281-a6f2-438f-a8fb-6ff851a4de77","Type":"ContainerStarted","Data":"32c47b551bd8bdffc46599ac3209e94d4d5dd2912629f8f90c298740e2f6c335"}
Jul 17 18:33:35.990641 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.990606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:35.991352 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:35.991327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ltlgb"
Jul 17 18:33:36.010213 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:36.010165    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mbqt6" podStartSLOduration=6.176691859 podStartE2EDuration="25.010149004s" totalImagesPullingTime="18.833457145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:35.871352194 +0000 UTC m=+24.782178968" watchObservedRunningTime="2026-07-17 18:33:36.010149004 +0000 UTC m=+24.920975764"
Jul 17 18:33:36.689367 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:36.689109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:36.689516 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:36.689160    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:36.689516 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:36.689417    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:36.689516 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:36.689475    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:36.853942 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:36.853899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx" event={"ID":"96fccbcb-c7f6-4dd9-acb4-213de051fb7e","Type":"ContainerStarted","Data":"28e6167442fc80e31bc890886118ef029e8357ee970d9ff565a8cc584ce4b12e"}
Jul 17 18:33:36.857418 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:36.857392    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:33:36.857838 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:36.857813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"0dbe77c94274e14f8ab04cd898ebf3c8c0920c74278a6f1b6a218ee620842105"}
Jul 17 18:33:36.883735 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:36.883684    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dp9sx" podStartSLOduration=4.807529779 podStartE2EDuration="25.883667801s" totalImagesPullingTime="21.076138022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:36.883293284 +0000 UTC m=+25.794120048" watchObservedRunningTime="2026-07-17 18:33:36.883667801 +0000 UTC m=+25.794494581"
Jul 17 18:33:37.689048 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:37.688996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:37.689274 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:37.689192    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:38.689055 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:38.689021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:38.689806 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:38.689026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:38.689806 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:38.689160    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:38.689806 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:38.689229    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:39.689291 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:39.689073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:39.689884 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:39.689373    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:39.867419 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:39.867379    2563 generic.go:356] "Generic (PLEG): container finished" podID="564a83b7-e7d0-48fc-b64c-ad02f0b688a8" containerID="3da0f3a27b3883a9aa031bf3c13d2918dc73cc54f5997cef02219d56ebd87334" exitCode=0
Jul 17 18:33:39.867557 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:39.867464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerDied","Data":"3da0f3a27b3883a9aa031bf3c13d2918dc73cc54f5997cef02219d56ebd87334"}
Jul 17 18:33:39.871016 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:39.870999    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:33:39.871426 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:39.871398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"971dbc74466a9904bd86717651ec3b1878fcda58f49c7a857bac8dc898641af0"}
Jul 17 18:33:39.871650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:39.871635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:39.871811 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:39.871796    2563 scope.go:117] "RemoveContainer" containerID="9da2a91696835731f70639f0f06036b6c10be704d593009afc8ef04eb8f3c146"
Jul 17 18:33:39.888547 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:39.888520    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:40.688811 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.688781    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:40.688986 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:40.688900    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:40.688986 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.688950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:40.689106 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:40.689042    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:40.875929 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.875896    2563 generic.go:356] "Generic (PLEG): container finished" podID="564a83b7-e7d0-48fc-b64c-ad02f0b688a8" containerID="8cb9426edd85ded82fc65240a0d06c770649d33b174172a3d84cab1fd5df9c2c" exitCode=0
Jul 17 18:33:40.876447 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.875977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerDied","Data":"8cb9426edd85ded82fc65240a0d06c770649d33b174172a3d84cab1fd5df9c2c"}
Jul 17 18:33:40.880671 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.880653    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:33:40.881052 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.881028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" event={"ID":"eb5d4daf-45c2-42f2-92fc-21c7e1e30083","Type":"ContainerStarted","Data":"3d20d5a8845e91864c70dd1c470f6b062693914f465c4bff1e0bba49a34e1229"}
Jul 17 18:33:40.881358 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.881334    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:40.881436 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.881386    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:40.897214 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.897185    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:33:40.946403 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:40.946347    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg" podStartSLOduration=11.058785425 podStartE2EDuration="29.946330894s" totalImagesPullingTime="18.887545469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:40.946218596 +0000 UTC m=+29.857045353" watchObservedRunningTime="2026-07-17 18:33:40.946330894 +0000 UTC m=+29.857157655"
Jul 17 18:33:41.334225 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:41.333912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t88vb"]
Jul 17 18:33:41.334361 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:41.334315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:41.334435 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:41.334410    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:41.337675 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:41.336661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hclhl"]
Jul 17 18:33:41.337675 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:41.336766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:41.337675 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:41.336853    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:41.337675 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:41.337455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hmfvz"]
Jul 17 18:33:41.337675 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:41.337563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:41.338065 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:41.338030    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:41.885350 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:41.885317    2563 generic.go:356] "Generic (PLEG): container finished" podID="564a83b7-e7d0-48fc-b64c-ad02f0b688a8" containerID="317eac7ba627ab689a0405dfc4e31551c33c4d1dbdb44fdf2b4033a889aec92a" exitCode=0
Jul 17 18:33:41.885722 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:41.885406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerDied","Data":"317eac7ba627ab689a0405dfc4e31551c33c4d1dbdb44fdf2b4033a889aec92a"}
Jul 17 18:33:42.688654 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:42.688621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:42.688824 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:42.688640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:42.688824 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:42.688738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:42.688954 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:42.688851    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:42.951359 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:42.950640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:43.123404 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:43.123347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:43.689270 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:43.689222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:43.689504 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:43.689371    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:44.688594 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:44.688557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:44.689161 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:44.688694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t88vb" podUID="f0baa97c-7dca-43b5-b52d-680c32593d70"
Jul 17 18:33:44.689161 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:44.688743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:44.689161 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:44.688876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:33:45.271040 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:45.270999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:45.271245 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.271196    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:45.271301 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.271280    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:17.271258935 +0000 UTC m=+66.182085675 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:45.372343 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:45.372295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:45.372532 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.372474    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:45.372595 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.372546    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret podName:f0baa97c-7dca-43b5-b52d-680c32593d70 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:17.372530754 +0000 UTC m=+66.283357503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret") pod "global-pull-secret-syncer-t88vb" (UID: "f0baa97c-7dca-43b5-b52d-680c32593d70") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:45.473762 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:45.473724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:45.473955 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.473873    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:45.473955 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.473892    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:45.473955 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.473901    2563 projected.go:196] Error preparing data for projected volume kube-api-access-2st2z for pod openshift-network-diagnostics/network-check-target-hclhl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:45.474113 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.473972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z podName:2737d09a-42b8-481d-8fae-378b80ca56ca nodeName:}" failed. No retries permitted until 2026-07-17 18:34:17.473952636 +0000 UTC m=+66.384779394 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2st2z" (UniqueName: "kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z") pod "network-check-target-hclhl" (UID: "2737d09a-42b8-481d-8fae-378b80ca56ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:45.689402 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:45.689363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:45.689933 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:45.689494    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hclhl" podUID="2737d09a-42b8-481d-8fae-378b80ca56ca"
Jul 17 18:33:46.048760 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.048690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:46.204448 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.204415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:46.450796 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.450766    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-130.ec2.internal" event="NodeReady"
Jul 17 18:33:46.450968 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.450958    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 18:33:46.523461 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.523435    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7dccw"]
Jul 17 18:33:46.540437 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.540410    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9tqqs"]
Jul 17 18:33:46.540549 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.540535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.545222 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.545203    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 18:33:46.545294 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.545206    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 18:33:46.545294 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.545288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz2rf\""
Jul 17 18:33:46.556760 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.556736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tqqs"
Jul 17 18:33:46.558319 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.558286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tqqs"]
Jul 17 18:33:46.559389 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.559369    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lzjfd\""
Jul 17 18:33:46.559512 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.559457    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 18:33:46.559965 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.559948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 18:33:46.560657 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.560639    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 18:33:46.564160 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.564088    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7dccw"]
Jul 17 18:33:46.683511 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.683476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.683704 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.683531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00f4c0f-381e-4dac-8330-c769a42e1b88-tmp-dir\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.683704 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.683554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nn4\" (UniqueName: \"kubernetes.io/projected/df37e276-43e7-41eb-88e6-4f78efcbcd7d-kube-api-access-89nn4\") pod \"ingress-canary-9tqqs\" (UID: \"df37e276-43e7-41eb-88e6-4f78efcbcd7d\") " pod="openshift-ingress-canary/ingress-canary-9tqqs"
Jul 17 18:33:46.683704 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.683586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f00f4c0f-381e-4dac-8330-c769a42e1b88-config-volume\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.683704 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.683636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxlf\" (UniqueName: \"kubernetes.io/projected/f00f4c0f-381e-4dac-8330-c769a42e1b88-kube-api-access-qmxlf\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.683704 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.683670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df37e276-43e7-41eb-88e6-4f78efcbcd7d-cert\") pod \"ingress-canary-9tqqs\" (UID: \"df37e276-43e7-41eb-88e6-4f78efcbcd7d\") " pod="openshift-ingress-canary/ingress-canary-9tqqs"
Jul 17 18:33:46.688540 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.688520    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:33:46.688698 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.688682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:33:46.694804 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.694600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gbnj9\""
Jul 17 18:33:46.694804 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.694660    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 18:33:46.695253 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.695011    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 18:33:46.784413 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.784379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.784595 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.784580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00f4c0f-381e-4dac-8330-c769a42e1b88-tmp-dir\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.784661 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.784612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nn4\" (UniqueName: \"kubernetes.io/projected/df37e276-43e7-41eb-88e6-4f78efcbcd7d-kube-api-access-89nn4\") pod \"ingress-canary-9tqqs\" (UID: \"df37e276-43e7-41eb-88e6-4f78efcbcd7d\") " pod="openshift-ingress-canary/ingress-canary-9tqqs"
Jul 17 18:33:46.784661 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.784649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f00f4c0f-381e-4dac-8330-c769a42e1b88-config-volume\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.784763 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.784671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxlf\" (UniqueName: \"kubernetes.io/projected/f00f4c0f-381e-4dac-8330-c769a42e1b88-kube-api-access-qmxlf\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.784763 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:46.784691    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:46.784763 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:46.784753    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:47.284732153 +0000 UTC m=+36.195558907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:46.784897 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.784693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df37e276-43e7-41eb-88e6-4f78efcbcd7d-cert\") pod \"ingress-canary-9tqqs\" (UID: \"df37e276-43e7-41eb-88e6-4f78efcbcd7d\") " pod="openshift-ingress-canary/ingress-canary-9tqqs"
Jul 17 18:33:46.784976 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.784957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00f4c0f-381e-4dac-8330-c769a42e1b88-tmp-dir\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.785266 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.785250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f00f4c0f-381e-4dac-8330-c769a42e1b88-config-volume\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.789050 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.789034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df37e276-43e7-41eb-88e6-4f78efcbcd7d-cert\") pod \"ingress-canary-9tqqs\" (UID: \"df37e276-43e7-41eb-88e6-4f78efcbcd7d\") " pod="openshift-ingress-canary/ingress-canary-9tqqs"
Jul 17 18:33:46.814154 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.814101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nn4\" (UniqueName: \"kubernetes.io/projected/df37e276-43e7-41eb-88e6-4f78efcbcd7d-kube-api-access-89nn4\") pod \"ingress-canary-9tqqs\" (UID: \"df37e276-43e7-41eb-88e6-4f78efcbcd7d\") " pod="openshift-ingress-canary/ingress-canary-9tqqs"
Jul 17 18:33:46.816647 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.816618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxlf\" (UniqueName: \"kubernetes.io/projected/f00f4c0f-381e-4dac-8330-c769a42e1b88-kube-api-access-qmxlf\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:46.899434 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.899230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tqqs"
Jul 17 18:33:46.900198 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.900175    2563 generic.go:356] "Generic (PLEG): container finished" podID="564a83b7-e7d0-48fc-b64c-ad02f0b688a8" containerID="0c5c7688fb7f31e93484ff6e8820c4388152a59461692a4b65b081ecbbc87799" exitCode=0
Jul 17 18:33:46.900304 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:46.900232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerDied","Data":"0c5c7688fb7f31e93484ff6e8820c4388152a59461692a4b65b081ecbbc87799"}
Jul 17 18:33:47.142738 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.142698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tqqs"]
Jul 17 18:33:47.148163 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:47.148138    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf37e276_43e7_41eb_88e6_4f78efcbcd7d.slice/crio-5850fb264829a7d3db92c57485204708466458e06f77b14503313e724194a733 WatchSource:0}: Error finding container 5850fb264829a7d3db92c57485204708466458e06f77b14503313e724194a733: Status 404 returned error can't find the container with id 5850fb264829a7d3db92c57485204708466458e06f77b14503313e724194a733
Jul 17 18:33:47.288381 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.288340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:47.288549 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:47.288483    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:47.288549 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:47.288548    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:48.288531382 +0000 UTC m=+37.199358124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:47.640364 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.640273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:47.689428 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.689392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:33:47.694111 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.694085    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 18:33:47.694280 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.694147    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-st5cn\""
Jul 17 18:33:47.694424 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.694408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 18:33:47.825751 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.825716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:47.906666 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.906568    2563 generic.go:356] "Generic (PLEG): container finished" podID="564a83b7-e7d0-48fc-b64c-ad02f0b688a8" containerID="740254925e382f40b223f9001abcbd9e98731db13c3fad31fa155b06c044ccee" exitCode=0
Jul 17 18:33:47.906822 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.906721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerDied","Data":"740254925e382f40b223f9001abcbd9e98731db13c3fad31fa155b06c044ccee"}
Jul 17 18:33:47.908140 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:47.907976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tqqs" event={"ID":"df37e276-43e7-41eb-88e6-4f78efcbcd7d","Type":"ContainerStarted","Data":"5850fb264829a7d3db92c57485204708466458e06f77b14503313e724194a733"}
Jul 17 18:33:48.028593 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:48.028557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:48.120430 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:48.120381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:48.295240 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:48.295152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:48.295414 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:48.295328    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:48.295414 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:48.295393    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:50.295375131 +0000 UTC m=+39.206201870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:48.915411 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:48.915188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" event={"ID":"564a83b7-e7d0-48fc-b64c-ad02f0b688a8","Type":"ContainerStarted","Data":"22a3cbcdb6276f5718d78af1fcced127b4b2b5e780545c145fed06463d71de8d"}
Jul 17 18:33:48.947306 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:48.947254    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mqfcl" podStartSLOduration=9.235473592 podStartE2EDuration="37.947236566s" totalImagesPullingTime="28.711762974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:48.946991994 +0000 UTC m=+37.857818757" watchObservedRunningTime="2026-07-17 18:33:48.947236566 +0000 UTC m=+37.858063396"
Jul 17 18:33:49.144104 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:49.144067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:49.283370 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:49.283284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:49.919330 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:49.919290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tqqs" event={"ID":"df37e276-43e7-41eb-88e6-4f78efcbcd7d","Type":"ContainerStarted","Data":"8f4dfe1f81bde97768d29ed563242bb61143bb9cf3f6d1a83284e144b6fc7eac"}
Jul 17 18:33:49.953294 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:49.953237    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9tqqs" podStartSLOduration=1.509202582 podStartE2EDuration="3.953221164s" totalImagesPullingTime="2.444018582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:49.95034182 +0000 UTC m=+38.861168599" watchObservedRunningTime="2026-07-17 18:33:49.953221164 +0000 UTC m=+38.864047939"
Jul 17 18:33:50.170764 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:50.170669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:50.265012 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:50.264969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:50.310374 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:50.310343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:50.310531 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:50.310513    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:50.310669 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:50.310595    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:54.310573191 +0000 UTC m=+43.221399938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:51.903741 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:51.903710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:52.034249 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:52.034210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:33:54.338674 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:54.338624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:33:54.339068 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:54.338799    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:54.339068 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:33:54.338873    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:02.338856309 +0000 UTC m=+51.249683048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:58.783372 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.783331    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"]
Jul 17 18:33:58.798468 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.798447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.812284 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.812263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 18:33:58.821157 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.821139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 18:33:58.821301 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.821278    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 17 18:33:58.821410 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.821301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 17 18:33:58.821410 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.821377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 18:33:58.821615 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.821599    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 17 18:33:58.822055 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.822040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 17 18:33:58.838942 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.838915    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"]
Jul 17 18:33:58.871271 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.871240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-hub\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.871271 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.871273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzx8k\" (UniqueName: \"kubernetes.io/projected/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-kube-api-access-mzx8k\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.871475 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.871300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.871475 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.871374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-ca\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.871552 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.871475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.871552 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.871512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.972373 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.972311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.972519 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.972396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.972519 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.972452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-hub\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.972622 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.972578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzx8k\" (UniqueName: \"kubernetes.io/projected/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-kube-api-access-mzx8k\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.972622 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.972616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.972707 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.972658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-ca\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.973174 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.973149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.976721 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.976696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.976721 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.976709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-ca\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.976850 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.976837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-hub\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.976915 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.976898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:58.982621 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:58.982602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzx8k\" (UniqueName: \"kubernetes.io/projected/7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e-kube-api-access-mzx8k\") pod \"cluster-proxy-proxy-agent-65554f86c6-xwzvg\" (UID: \"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:59.122278 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:59.122244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:33:59.290400 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:59.290357    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"]
Jul 17 18:33:59.294559 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:33:59.294532    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7809eb55_cc3b_4b9a_9dbb_c59f4827ea6e.slice/crio-c07a2fc5e4a082a7b0baf62f2d85bb647c8f04e2535cdc31e8aa7b0bae70bba3 WatchSource:0}: Error finding container c07a2fc5e4a082a7b0baf62f2d85bb647c8f04e2535cdc31e8aa7b0bae70bba3: Status 404 returned error can't find the container with id c07a2fc5e4a082a7b0baf62f2d85bb647c8f04e2535cdc31e8aa7b0bae70bba3
Jul 17 18:33:59.943626 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:33:59.943593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" event={"ID":"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e","Type":"ContainerStarted","Data":"c07a2fc5e4a082a7b0baf62f2d85bb647c8f04e2535cdc31e8aa7b0bae70bba3"}
Jul 17 18:34:02.401177 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:02.401137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:34:02.401596 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:34:02.401227    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:34:02.401596 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:34:02.401295    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:18.401279938 +0000 UTC m=+67.312106697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:34:05.218175 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:05.218138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:05.332744 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:05.332709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:08.966858 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:08.966822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" event={"ID":"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e","Type":"ContainerStarted","Data":"b66709576e9099d7fc5fe8b07fc7f4c3c6dc31ec2704434502528dc631f04262"}
Jul 17 18:34:08.966858 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:08.966861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" event={"ID":"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e","Type":"ContainerStarted","Data":"383d0a0f2af2562fc0154dc1bef681ba3ed9668f788991fe0d7b9458c0efe719"}
Jul 17 18:34:08.967397 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:08.966874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" event={"ID":"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e","Type":"ContainerStarted","Data":"336767fe0bb3022122a0c6eaf677000ffe337388d7c387238d14664ff9bc781f"}
Jul 17 18:34:11.549480 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:11.549443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:11.685650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:11.685611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:12.898780 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:12.898751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g49xg"
Jul 17 18:34:12.952700 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:12.952628    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" podStartSLOduration=6.103020593 podStartE2EDuration="14.952605077s" totalImagesPullingTime="8.849584484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:09.010352713 +0000 UTC m=+57.921179493" watchObservedRunningTime="2026-07-17 18:34:12.952605077 +0000 UTC m=+61.863431838"
Jul 17 18:34:13.128445 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:13.128399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:13.223163 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:13.223067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:16.379079 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:16.379041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:17.314135 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.314078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:34:17.319640 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.319621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 18:34:17.324920 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:34:17.324901    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 18:34:17.324990 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:34:17.324980    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:35:21.324963277 +0000 UTC m=+130.235790015 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : secret "metrics-daemon-secret" not found
Jul 17 18:34:17.415367 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.415327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:34:17.418180 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.418162    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 18:34:17.428851 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.428829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0baa97c-7dca-43b5-b52d-680c32593d70-original-pull-secret\") pod \"global-pull-secret-syncer-t88vb\" (UID: \"f0baa97c-7dca-43b5-b52d-680c32593d70\") " pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:34:17.516175 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.516132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:34:17.520468 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.520451    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 18:34:17.530958 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.530937    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 18:34:17.539758 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.539737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2st2z\" (UniqueName: \"kubernetes.io/projected/2737d09a-42b8-481d-8fae-378b80ca56ca-kube-api-access-2st2z\") pod \"network-check-target-hclhl\" (UID: \"2737d09a-42b8-481d-8fae-378b80ca56ca\") " pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:34:17.602747 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.602713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t88vb"
Jul 17 18:34:17.703358 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.703333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-st5cn\""
Jul 17 18:34:17.711427 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.711400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:34:17.755239 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.755043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t88vb"]
Jul 17 18:34:17.758901 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:34:17.758876    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0baa97c_7dca_43b5_b52d_680c32593d70.slice/crio-2861c038922bfb3d3705648a7105af9907e954aad211b56f035f6300d1faba19 WatchSource:0}: Error finding container 2861c038922bfb3d3705648a7105af9907e954aad211b56f035f6300d1faba19: Status 404 returned error can't find the container with id 2861c038922bfb3d3705648a7105af9907e954aad211b56f035f6300d1faba19
Jul 17 18:34:17.875091 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.875013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hclhl"]
Jul 17 18:34:17.878240 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:34:17.878216    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2737d09a_42b8_481d_8fae_378b80ca56ca.slice/crio-caae483b5f7bd4b0e5575693495b205aa49cef7dfd6317a270f58b321279b6b1 WatchSource:0}: Error finding container caae483b5f7bd4b0e5575693495b205aa49cef7dfd6317a270f58b321279b6b1: Status 404 returned error can't find the container with id caae483b5f7bd4b0e5575693495b205aa49cef7dfd6317a270f58b321279b6b1
Jul 17 18:34:17.987640 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.987603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hclhl" event={"ID":"2737d09a-42b8-481d-8fae-378b80ca56ca","Type":"ContainerStarted","Data":"caae483b5f7bd4b0e5575693495b205aa49cef7dfd6317a270f58b321279b6b1"}
Jul 17 18:34:17.988749 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:17.988726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t88vb" event={"ID":"f0baa97c-7dca-43b5-b52d-680c32593d70","Type":"ContainerStarted","Data":"2861c038922bfb3d3705648a7105af9907e954aad211b56f035f6300d1faba19"}
Jul 17 18:34:18.425682 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:18.425634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:34:18.426223 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:34:18.425836    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:34:18.426223 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:34:18.425891    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:50.425877002 +0000 UTC m=+99.336703744 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:34:19.074619 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:19.074574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:19.181270 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:19.181231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:34:23.004998 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:23.004964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hclhl" event={"ID":"2737d09a-42b8-481d-8fae-378b80ca56ca","Type":"ContainerStarted","Data":"fdfd2e903ca6b4e963462cfb8eaaae1c2775c56b85055d09fe7c34dc453626ec"}
Jul 17 18:34:23.005435 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:23.005084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:34:23.006425 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:23.006405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t88vb" event={"ID":"f0baa97c-7dca-43b5-b52d-680c32593d70","Type":"ContainerStarted","Data":"d442d647d0c016b0a5d538456b5a216890f110199dc70fad24409c6814cbf55c"}
Jul 17 18:34:23.026316 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:23.026268    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hclhl" podStartSLOduration=67.704267947 podStartE2EDuration="1m12.026253093s" totalImagesPullingTime="4.321985146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:23.025188677 +0000 UTC m=+71.936015439" watchObservedRunningTime="2026-07-17 18:34:23.026253093 +0000 UTC m=+71.937079851"
Jul 17 18:34:50.452233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:50.452067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:34:50.452731 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:34:50.452240    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:34:50.452731 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:34:50.452325    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:35:54.452308152 +0000 UTC m=+163.363134892 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:34:54.011987 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:54.011957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hclhl"
Jul 17 18:34:54.032531 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:34:54.032385    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t88vb" podStartSLOduration=97.587595846 podStartE2EDuration="1m42.032347688s" totalImagesPullingTime="4.444751842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:23.064882014 +0000 UTC m=+71.975708792" watchObservedRunningTime="2026-07-17 18:34:54.032347688 +0000 UTC m=+102.943174447"
Jul 17 18:35:05.452896 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:35:05.452853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:35:21.375711 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:35:21.375653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:35:21.376247 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:35:21.375818    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 18:35:21.376247 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:35:21.375892    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:23.375873372 +0000 UTC m=+252.286700114 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : secret "metrics-daemon-secret" not found
Jul 17 18:35:49.594934 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:35:49.594889    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7dccw" podUID="f00f4c0f-381e-4dac-8330-c769a42e1b88"
Jul 17 18:35:49.697545 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:35:49.697508    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:35:50.246604 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:35:50.246572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7dccw"
Jul 17 18:35:54.517685 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:35:54.517646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:35:54.518097 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:35:54.517791    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:35:54.518097 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:35:54.517875    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls podName:f00f4c0f-381e-4dac-8330-c769a42e1b88 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:56.517855921 +0000 UTC m=+285.428682680 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls") pod "dns-default-7dccw" (UID: "f00f4c0f-381e-4dac-8330-c769a42e1b88") : secret "dns-default-metrics-tls" not found
Jul 17 18:36:04.689579 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:36:04.689487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:36:26.875912 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:36:26.875870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:16.390587 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.390539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:16.769222 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.769109    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6564dc85c9-dgznl"]
Jul 17 18:37:16.771233 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.771214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.777917 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.777895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 18:37:16.781760 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.781736    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 18:37:16.782280 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.782260    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h6x9z\""
Jul 17 18:37:16.790499 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.790476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 18:37:16.793284 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.793264    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 18:37:16.802244 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.802209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6564dc85c9-dgznl"]
Jul 17 18:37:16.878783 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.878749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-ca-trust-extracted\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.878783 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.878790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-installation-pull-secrets\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.879027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.878818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.879027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.878879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-trusted-ca\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.879027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.878917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-bound-sa-token\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.879027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.878967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-image-registry-private-configuration\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.879027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.878995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmzm\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-kube-api-access-7lmzm\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.879027 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.879018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-certificates\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979340 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-trusted-ca\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979340 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-bound-sa-token\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979533 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-image-registry-private-configuration\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979533 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmzm\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-kube-api-access-7lmzm\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979637 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-certificates\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979637 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-ca-trust-extracted\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979637 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-installation-pull-secrets\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979798 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.979798 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:16.979754    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:37:16.979798 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:16.979769    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6564dc85c9-dgznl: secret "image-registry-tls" not found
Jul 17 18:37:16.979942 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:16.979831    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls podName:da3948d5-fbd0-43ed-8e2e-bf1961e94fb6 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:17.479810964 +0000 UTC m=+246.390637708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls") pod "image-registry-6564dc85c9-dgznl" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6") : secret "image-registry-tls" not found
Jul 17 18:37:16.979990 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.979957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-ca-trust-extracted\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.980444 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.980367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-certificates\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.980563 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.980509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-trusted-ca\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.981942 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.981923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-image-registry-private-configuration\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.982045 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.981996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-installation-pull-secrets\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.988289 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.988266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-bound-sa-token\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:16.988990 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:16.988967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmzm\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-kube-api-access-7lmzm\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:17.483889 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:17.483842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:17.484319 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:17.484008    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:37:17.484319 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:17.484029    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6564dc85c9-dgznl: secret "image-registry-tls" not found
Jul 17 18:37:17.484319 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:17.484084    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls podName:da3948d5-fbd0-43ed-8e2e-bf1961e94fb6 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:18.48406841 +0000 UTC m=+247.394895150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls") pod "image-registry-6564dc85c9-dgznl" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6") : secret "image-registry-tls" not found
Jul 17 18:37:17.931109 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:17.931073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:18.491906 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:18.491872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:18.492365 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:18.492047    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:37:18.492365 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:18.492072    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6564dc85c9-dgznl: secret "image-registry-tls" not found
Jul 17 18:37:18.492365 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:18.492154    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls podName:da3948d5-fbd0-43ed-8e2e-bf1961e94fb6 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:20.492133218 +0000 UTC m=+249.402959970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls") pod "image-registry-6564dc85c9-dgznl" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6") : secret "image-registry-tls" not found
Jul 17 18:37:19.558015 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:19.557979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:19.633585 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:19.633548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:20.509309 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:20.509265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:20.509493 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:20.509415    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:37:20.509493 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:20.509433    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6564dc85c9-dgznl: secret "image-registry-tls" not found
Jul 17 18:37:20.509580 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:20.509498    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls podName:da3948d5-fbd0-43ed-8e2e-bf1961e94fb6 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:24.509481738 +0000 UTC m=+253.420308477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls") pod "image-registry-6564dc85c9-dgznl" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6") : secret "image-registry-tls" not found
Jul 17 18:37:21.031821 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.031789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:21.145555 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.145522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:21.247886 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.247847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:21.403784 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.403753    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sr4bl"]
Jul 17 18:37:21.406356 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.406340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.408491 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.408451    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 18:37:21.408614 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.408541    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 18:37:21.408696 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.408682    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 18:37:21.409531 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.409511    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 18:37:21.409531 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.409524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvlrf\""
Jul 17 18:37:21.427258 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.427226    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sr4bl"]
Jul 17 18:37:21.451334 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.451306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:21.518466 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.518425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47x9\" (UniqueName: \"kubernetes.io/projected/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-api-access-z47x9\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.518466 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.518462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c50c47d-872f-4c0f-8784-793d22823ecd-crio-socket\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.518664 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.518527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.518664 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.518559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c50c47d-872f-4c0f-8784-793d22823ecd-data-volume\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.518664 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.518575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.542160 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.542105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:21.618929 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.618890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47x9\" (UniqueName: \"kubernetes.io/projected/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-api-access-z47x9\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.618929 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.618925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c50c47d-872f-4c0f-8784-793d22823ecd-crio-socket\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.619239 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.618976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.619239 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.619018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c50c47d-872f-4c0f-8784-793d22823ecd-data-volume\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.619239 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.619030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c50c47d-872f-4c0f-8784-793d22823ecd-crio-socket\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.619239 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.619037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.619239 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:21.619091    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:21.619239 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:21.619160    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:21.619239 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:21.619177    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:22.119155298 +0000 UTC m=+251.029982056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:21.619239 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:21.619232    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:22.11921481 +0000 UTC m=+251.030041552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:21.619597 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.619416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c50c47d-872f-4c0f-8784-793d22823ecd-data-volume\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:21.630840 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:21.630811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47x9\" (UniqueName: \"kubernetes.io/projected/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-api-access-z47x9\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:22.058640 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:22.058604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:22.123867 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:22.123827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:22.124059 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:22.123895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:22.124059 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:22.123950    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:22.124059 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:22.123988    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:22.124059 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:22.124015    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:23.12399838 +0000 UTC m=+252.034825119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:22.124059 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:22.124045    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:23.124032448 +0000 UTC m=+252.034859188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:22.162346 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:22.162313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:22.502969 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:22.502933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:22.604624 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:22.604589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:22.788608 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:22.788501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:22.888481 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:22.888438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:23.039172 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:23.039034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:23.132904 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:23.132871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:23.133330 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:23.132926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:23.133330 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:23.133008    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:23.133330 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:23.133044    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:23.133330 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:23.133093    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:25.133069264 +0000 UTC m=+254.043896003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:23.133330 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:23.133112    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:25.133103005 +0000 UTC m=+254.043929751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:23.143652 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:23.143609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:23.434884 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:23.434847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:37:23.435066 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:23.434983    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 18:37:23.435066 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:23.435040    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs podName:e0553119-5c66-4b8e-9e4b-c68c8f06af08 nodeName:}" failed. No retries permitted until 2026-07-17 18:39:25.435024259 +0000 UTC m=+374.345850998 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs") pod "network-metrics-daemon-hmfvz" (UID: "e0553119-5c66-4b8e-9e4b-c68c8f06af08") : secret "metrics-daemon-secret" not found
Jul 17 18:37:23.992545 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:23.992516    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dlf6l_7d3069f6-5dc2-40cd-9459-917d4358b97c/dns-node-resolver/0.log"
Jul 17 18:37:24.477235 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:24.477201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:24.544143 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:24.544081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:24.544326 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:24.544245    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:37:24.544326 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:24.544262    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6564dc85c9-dgznl: secret "image-registry-tls" not found
Jul 17 18:37:24.544326 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:24.544316    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls podName:da3948d5-fbd0-43ed-8e2e-bf1961e94fb6 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:32.544300793 +0000 UTC m=+261.455127532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls") pod "image-registry-6564dc85c9-dgznl" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6") : secret "image-registry-tls" not found
Jul 17 18:37:24.579579 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:24.579549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:24.994989 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:24.994966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lsgfn_37c764c7-9cd9-40cd-8363-87505d4d1df3/node-ca/0.log"
Jul 17 18:37:25.147574 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:25.147522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:25.147748 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:25.147603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:25.147748 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:25.147651    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:25.147748 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:25.147708    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:25.147748 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:25.147718    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:29.147698634 +0000 UTC m=+258.058525376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:25.147748 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:25.147741    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:29.147730861 +0000 UTC m=+258.058557604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:25.500199 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:25.500157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:25.793533 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:25.793451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tqqs_df37e276-43e7-41eb-88e6-4f78efcbcd7d/serve-healthcheck-canary/0.log"
Jul 17 18:37:26.316516 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:26.316481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:26.487868 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:26.487836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:26.608364 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:26.608324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:26.721490 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:26.721458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:26.824950 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:26.824916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:26.986864 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:26.986779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:27.184880 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:27.184844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:27.564090 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:27.564059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:28.388631 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:28.388597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:28.661260 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:28.661173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:28.984061 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:28.983972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:29.107460 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:29.107424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:29.183578 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:29.183539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:29.183758 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:29.183592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:29.183758 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:29.183662    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:29.183758 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:29.183700    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:29.183758 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:29.183727    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:37.183711923 +0000 UTC m=+266.094538665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:29.183758 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:29.183745    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls podName:6c50c47d-872f-4c0f-8784-793d22823ecd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:37.183733907 +0000 UTC m=+266.094560646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr4bl" (UID: "6c50c47d-872f-4c0f-8784-793d22823ecd") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:29.309935 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:29.309841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:29.409713 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:29.409678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:30.192837 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:30.192792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:30.294395 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:30.294359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:30.399318 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:30.399266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:30.787439 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:30.787402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:30.985230 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:30.985194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:31.190245 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:31.190211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:31.387524 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:31.387490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:32.590095 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:32.590064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:32.608724 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:32.608693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:32.611124 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:32.611098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"image-registry-6564dc85c9-dgznl\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") " pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:32.680261 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:32.680218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:32.786925 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:32.786889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:32.818490 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:32.818452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6564dc85c9-dgznl"]
Jul 17 18:37:32.824918 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:37:32.824890    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3948d5_fbd0_43ed_8e2e_bf1961e94fb6.slice/crio-49b9d9b06a77ba7b1a755180800c4b6d8f3cdca4c8adf020ed834b4d4d528155 WatchSource:0}: Error finding container 49b9d9b06a77ba7b1a755180800c4b6d8f3cdca4c8adf020ed834b4d4d528155: Status 404 returned error can't find the container with id 49b9d9b06a77ba7b1a755180800c4b6d8f3cdca4c8adf020ed834b4d4d528155
Jul 17 18:37:32.888559 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:32.888531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:32.991535 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:32.991493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:33.198078 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:33.197984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:33.419161 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:33.419097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:33.534134 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:33.534039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl" event={"ID":"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6","Type":"ContainerStarted","Data":"ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c"}
Jul 17 18:37:33.534134 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:33.534086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl" event={"ID":"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6","Type":"ContainerStarted","Data":"49b9d9b06a77ba7b1a755180800c4b6d8f3cdca4c8adf020ed834b4d4d528155"}
Jul 17 18:37:33.534321 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:33.534137    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:37:33.563715 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:33.563660    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl" podStartSLOduration=17.56364462 podStartE2EDuration="17.56364462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:33.5630304 +0000 UTC m=+262.473857180" watchObservedRunningTime="2026-07-17 18:37:33.56364462 +0000 UTC m=+262.474471381"
Jul 17 18:37:34.173446 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:34.173410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:34.251679 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:34.251651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:34.590554 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:34.590455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:34.796710 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:34.796666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:35.229272 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:35.229242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:35.981327 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:35.981286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:36.194942 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:36.194911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:36.398632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:36.398602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:36.586625 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:36.586593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:37.044193 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.044161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:37.145553 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.145517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:37.245335 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.245297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:37.245485 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.245377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:37.245870 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.245853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c50c47d-872f-4c0f-8784-793d22823ecd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:37.247709 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.247694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c50c47d-872f-4c0f-8784-793d22823ecd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr4bl\" (UID: \"6c50c47d-872f-4c0f-8784-793d22823ecd\") " pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:37.315545 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.315449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sr4bl"
Jul 17 18:37:37.447371 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.447335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sr4bl"]
Jul 17 18:37:37.452074 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:37:37.452039    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c50c47d_872f_4c0f_8784_793d22823ecd.slice/crio-e8fb588120f0e8140a8fd3e64b0dd1976178395a7f74dd4d24408a0735c47814 WatchSource:0}: Error finding container e8fb588120f0e8140a8fd3e64b0dd1976178395a7f74dd4d24408a0735c47814: Status 404 returned error can't find the container with id e8fb588120f0e8140a8fd3e64b0dd1976178395a7f74dd4d24408a0735c47814
Jul 17 18:37:37.547070 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:37.547041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sr4bl" event={"ID":"6c50c47d-872f-4c0f-8784-793d22823ecd","Type":"ContainerStarted","Data":"e8fb588120f0e8140a8fd3e64b0dd1976178395a7f74dd4d24408a0735c47814"}
Jul 17 18:37:39.371035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:39.371000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:39.471190 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:39.471147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:39.653889 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:39.653796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:40.029763 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:40.029680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:40.106813 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:40.106777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:40.456371 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:40.456336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:40.533740 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:40.533691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:40.623270 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:40.623238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:40.713794 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:40.713688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:41.034685 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:41.034581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:41.240806 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:41.240770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:41.341590 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:41.341497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:41.442672 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:41.442636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:42.253812 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:42.253775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:42.354349 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:42.354317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:42.567087 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:42.567007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sr4bl" event={"ID":"6c50c47d-872f-4c0f-8784-793d22823ecd","Type":"ContainerStarted","Data":"befa46078fbde165aaa829530a72f435b576b37eeb2920ce1611431cd5614cc4"}
Jul 17 18:37:46.108799 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.108765    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6564dc85c9-dgznl"]
Jul 17 18:37:46.240945 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.240907    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b7d76964d-pht9r"]
Jul 17 18:37:46.243845 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.243828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.276287 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.276246    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b7d76964d-pht9r"]
Jul 17 18:37:46.315432 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.315392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-bound-sa-token\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.315432 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.315439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/199cb7af-c161-4c54-997f-8d8f8476c9a5-registry-certificates\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.315688 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.315457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-registry-tls\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.315688 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.315500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/199cb7af-c161-4c54-997f-8d8f8476c9a5-image-registry-private-configuration\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.315688 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.315544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/199cb7af-c161-4c54-997f-8d8f8476c9a5-trusted-ca\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.315688 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.315584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvlq\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-kube-api-access-4lvlq\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.315688 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.315610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/199cb7af-c161-4c54-997f-8d8f8476c9a5-installation-pull-secrets\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.315688 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.315642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/199cb7af-c161-4c54-997f-8d8f8476c9a5-ca-trust-extracted\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.325305 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.325270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:46.415963 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.415933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/199cb7af-c161-4c54-997f-8d8f8476c9a5-registry-certificates\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.415963 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.415967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-registry-tls\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.416271 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.416010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/199cb7af-c161-4c54-997f-8d8f8476c9a5-image-registry-private-configuration\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.416271 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.416030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/199cb7af-c161-4c54-997f-8d8f8476c9a5-trusted-ca\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.416271 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.416055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvlq\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-kube-api-access-4lvlq\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.416271 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.416081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/199cb7af-c161-4c54-997f-8d8f8476c9a5-installation-pull-secrets\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.416271 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.416113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/199cb7af-c161-4c54-997f-8d8f8476c9a5-ca-trust-extracted\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.416271 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.416184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-bound-sa-token\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.416693 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.416667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/199cb7af-c161-4c54-997f-8d8f8476c9a5-ca-trust-extracted\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.416935 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.416850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/199cb7af-c161-4c54-997f-8d8f8476c9a5-registry-certificates\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.417062 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.417039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/199cb7af-c161-4c54-997f-8d8f8476c9a5-trusted-ca\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.418668 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.418649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-registry-tls\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.419246 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.419055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/199cb7af-c161-4c54-997f-8d8f8476c9a5-image-registry-private-configuration\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.419246 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.419203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/199cb7af-c161-4c54-997f-8d8f8476c9a5-installation-pull-secrets\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.441021 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.440983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-bound-sa-token\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.441782 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.441751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvlq\" (UniqueName: \"kubernetes.io/projected/199cb7af-c161-4c54-997f-8d8f8476c9a5-kube-api-access-4lvlq\") pod \"image-registry-b7d76964d-pht9r\" (UID: \"199cb7af-c161-4c54-997f-8d8f8476c9a5\") " pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.461441 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.461407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:46.552645 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.552563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:46.582933 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.582894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sr4bl" event={"ID":"6c50c47d-872f-4c0f-8784-793d22823ecd","Type":"ContainerStarted","Data":"16800058f51db0ea5c56e488f6f27fcf0bc9efbaee35296bc42a42e34877ff72"}
Jul 17 18:37:46.719614 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:46.719574    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b7d76964d-pht9r"]
Jul 17 18:37:46.726243 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:37:46.726211    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199cb7af_c161_4c54_997f_8d8f8476c9a5.slice/crio-470d30e065096c4b47120f0936d0d21ae1595218ae29282908e82686fc65890b WatchSource:0}: Error finding container 470d30e065096c4b47120f0936d0d21ae1595218ae29282908e82686fc65890b: Status 404 returned error can't find the container with id 470d30e065096c4b47120f0936d0d21ae1595218ae29282908e82686fc65890b
Jul 17 18:37:47.587951 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:47.587908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b7d76964d-pht9r" event={"ID":"199cb7af-c161-4c54-997f-8d8f8476c9a5","Type":"ContainerStarted","Data":"6bab2accedd8769d2ef7e10a543f98836fb52b105c13b7ed6dba233287017e87"}
Jul 17 18:37:47.588468 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:47.587973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b7d76964d-pht9r" event={"ID":"199cb7af-c161-4c54-997f-8d8f8476c9a5","Type":"ContainerStarted","Data":"470d30e065096c4b47120f0936d0d21ae1595218ae29282908e82686fc65890b"}
Jul 17 18:37:47.588468 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:47.588031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:37:47.614429 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:47.614353    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b7d76964d-pht9r" podStartSLOduration=1.6143065970000001 podStartE2EDuration="1.614306597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:47.613441343 +0000 UTC m=+276.524268109" watchObservedRunningTime="2026-07-17 18:37:47.614306597 +0000 UTC m=+276.525133350"
Jul 17 18:37:48.001463 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:48.001414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:48.109510 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:48.109474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:49.598330 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:49.598288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sr4bl" event={"ID":"6c50c47d-872f-4c0f-8784-793d22823ecd","Type":"ContainerStarted","Data":"8d9546d16b8a44a18659154b3f7e720c54e2231149e485286b70daaaca30a9a6"}
Jul 17 18:37:49.629704 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:49.629644    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sr4bl" podStartSLOduration=16.690400784 podStartE2EDuration="28.629622334s" totalImagesPullingTime="11.93922155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:37:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:49.628257558 +0000 UTC m=+278.539084346" watchObservedRunningTime="2026-07-17 18:37:49.629622334 +0000 UTC m=+278.540449098"
Jul 17 18:37:51.737014 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:51.736963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:51.951600 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:51.951562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:52.757016 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:52.756972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:52.831195 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:52.831163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:53.248009 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:37:53.247966    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7dccw" podUID="f00f4c0f-381e-4dac-8330-c769a42e1b88"
Jul 17 18:37:53.609575 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:53.609546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7dccw"
Jul 17 18:37:54.620902 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:54.620871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:54.741652 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:54.741620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:54.831393 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:54.831355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:54.991199 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:54.991091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:56.114410 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:56.114376    2563 patch_prober.go:28] interesting pod/image-registry-6564dc85c9-dgznl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 18:37:56.114770 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:56.114426    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl" podUID="da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 18:37:56.594701 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:56.594669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:37:56.597164 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:56.597111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00f4c0f-381e-4dac-8330-c769a42e1b88-metrics-tls\") pod \"dns-default-7dccw\" (UID: \"f00f4c0f-381e-4dac-8330-c769a42e1b88\") " pod="openshift-dns/dns-default-7dccw"
Jul 17 18:37:56.612806 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:56.612781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz2rf\""
Jul 17 18:37:56.621181 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:56.621156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7dccw"
Jul 17 18:37:56.757701 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:56.757670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7dccw"]
Jul 17 18:37:56.761136 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:37:56.761090    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00f4c0f_381e_4dac_8330_c769a42e1b88.slice/crio-fb450fbca9bacbdcff9ce368ab02a8e6e13e7df55366d08192287165f16d04c7 WatchSource:0}: Error finding container fb450fbca9bacbdcff9ce368ab02a8e6e13e7df55366d08192287165f16d04c7: Status 404 returned error can't find the container with id fb450fbca9bacbdcff9ce368ab02a8e6e13e7df55366d08192287165f16d04c7
Jul 17 18:37:57.622136 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:57.622090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7dccw" event={"ID":"f00f4c0f-381e-4dac-8330-c769a42e1b88","Type":"ContainerStarted","Data":"fb450fbca9bacbdcff9ce368ab02a8e6e13e7df55366d08192287165f16d04c7"}
Jul 17 18:37:57.816113 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:57.816054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:58.020962 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:58.020888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:58.128299 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:58.128260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:58.626671 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:58.626646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7dccw" event={"ID":"f00f4c0f-381e-4dac-8330-c769a42e1b88","Type":"ContainerStarted","Data":"a5f015381ef6e90bb2ab195136d47c98122d38bde226574c3ae80edc50450e81"}
Jul 17 18:37:58.873599 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:58.873561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:58.978627 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:58.978588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:59.074611 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:59.074581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:37:59.631690 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:59.631650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7dccw" event={"ID":"f00f4c0f-381e-4dac-8330-c769a42e1b88","Type":"ContainerStarted","Data":"daed8c7919a028ebad272d342bb6e062dc42f7ef8be7252d8152b2b1a6dbc1b2"}
Jul 17 18:37:59.632149 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:59.631792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7dccw"
Jul 17 18:37:59.652260 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:37:59.652212    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7dccw" podStartSLOduration=251.890838668 podStartE2EDuration="4m13.652197069s" totalImagesPullingTime="1.761358401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:59.651770854 +0000 UTC m=+288.562597651" watchObservedRunningTime="2026-07-17 18:37:59.652197069 +0000 UTC m=+288.563023830"
Jul 17 18:38:00.323223 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:00.323179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:02.484848 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.484808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:02.541928 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.541895    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-frrn9"]
Jul 17 18:38:02.543911 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.543891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.549441 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.549414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 18:38:02.549587 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.549419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 18:38:02.549667 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.549649    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 18:38:02.549721 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.549654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 18:38:02.550224 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.550208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 18:38:02.550513 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.550498    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bkwv8\""
Jul 17 18:38:02.550590 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.550488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 18:38:02.605110 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.605075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:02.639073 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-root\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.639073 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-sys\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.639329 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-node-exporter-wtmp\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.639329 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.639329 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282366ea-d550-407f-a397-81dafba91392-metrics-client-ca\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.639329 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-tls\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.639329 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/282366ea-d550-407f-a397-81dafba91392-node-exporter-accelerators-collector-config\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.639490 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjrx\" (UniqueName: \"kubernetes.io/projected/282366ea-d550-407f-a397-81dafba91392-kube-api-access-7bjrx\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.639490 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.639387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/282366ea-d550-407f-a397-81dafba91392-node-exporter-textfile\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.703046 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.703011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:02.740099 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjrx\" (UniqueName: \"kubernetes.io/projected/282366ea-d550-407f-a397-81dafba91392-kube-api-access-7bjrx\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740099 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/282366ea-d550-407f-a397-81dafba91392-node-exporter-textfile\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740354 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-root\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740354 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-sys\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740354 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-root\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740354 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-sys\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740544 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-node-exporter-wtmp\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740544 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740544 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282366ea-d550-407f-a397-81dafba91392-metrics-client-ca\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740544 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-tls\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740739 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/282366ea-d550-407f-a397-81dafba91392-node-exporter-wtmp\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740739 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/282366ea-d550-407f-a397-81dafba91392-node-exporter-accelerators-collector-config\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740739 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.740467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/282366ea-d550-407f-a397-81dafba91392-node-exporter-textfile\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.740739 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:38:02.740698    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 18:38:02.740931 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:38:02.740765    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-tls podName:282366ea-d550-407f-a397-81dafba91392 nodeName:}" failed. No retries permitted until 2026-07-17 18:38:03.240745825 +0000 UTC m=+292.151572569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-tls") pod "node-exporter-frrn9" (UID: "282366ea-d550-407f-a397-81dafba91392") : secret "node-exporter-tls" not found
Jul 17 18:38:02.741035 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.741014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282366ea-d550-407f-a397-81dafba91392-metrics-client-ca\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.741231 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.741215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/282366ea-d550-407f-a397-81dafba91392-node-exporter-accelerators-collector-config\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.742825 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.742806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.759602 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.759570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjrx\" (UniqueName: \"kubernetes.io/projected/282366ea-d550-407f-a397-81dafba91392-kube-api-access-7bjrx\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:02.801410 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.801374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:02.904598 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:02.904564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:03.244244 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:03.244212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-tls\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:03.246664 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:03.246632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/282366ea-d550-407f-a397-81dafba91392-node-exporter-tls\") pod \"node-exporter-frrn9\" (UID: \"282366ea-d550-407f-a397-81dafba91392\") " pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:03.457158 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:03.457107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-frrn9"
Jul 17 18:38:03.467581 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:38:03.467537    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282366ea_d550_407f_a397_81dafba91392.slice/crio-4045f6ddfc917c8984df35fdff742e2e712b7331602be3b6f9c8d22935afab2c WatchSource:0}: Error finding container 4045f6ddfc917c8984df35fdff742e2e712b7331602be3b6f9c8d22935afab2c: Status 404 returned error can't find the container with id 4045f6ddfc917c8984df35fdff742e2e712b7331602be3b6f9c8d22935afab2c
Jul 17 18:38:03.498713 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:03.498621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:03.643453 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:03.643423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:03.645894 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:03.645867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frrn9" event={"ID":"282366ea-d550-407f-a397-81dafba91392","Type":"ContainerStarted","Data":"4045f6ddfc917c8984df35fdff742e2e712b7331602be3b6f9c8d22935afab2c"}
Jul 17 18:38:03.861563 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:03.861531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:04.162537 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:04.162455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:04.263180 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:04.263148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:04.362239 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:04.362192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:05.421044 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:05.421015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:05.508041 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:05.508011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:05.655454 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:05.655420    2563 generic.go:356] "Generic (PLEG): container finished" podID="282366ea-d550-407f-a397-81dafba91392" containerID="64374812584a074e4b25ead610bfa1c3b32dd889536e8b1a9d91625fe0bb742a" exitCode=0
Jul 17 18:38:05.655593 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:05.655499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frrn9" event={"ID":"282366ea-d550-407f-a397-81dafba91392","Type":"ContainerDied","Data":"64374812584a074e4b25ead610bfa1c3b32dd889536e8b1a9d91625fe0bb742a"}
Jul 17 18:38:05.739995 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:05.739961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:05.869111 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:05.869081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:06.113158 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:06.113111    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:38:06.557463 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:06.557372    2563 patch_prober.go:28] interesting pod/image-registry-b7d76964d-pht9r container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 18:38:06.557463 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:06.557432    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b7d76964d-pht9r" podUID="199cb7af-c161-4c54-997f-8d8f8476c9a5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 18:38:06.660901 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:06.660869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frrn9" event={"ID":"282366ea-d550-407f-a397-81dafba91392","Type":"ContainerStarted","Data":"4e0f42c19ab168415a91731c332cfd12bf4c58dcae009b04cd039db03d0875aa"}
Jul 17 18:38:06.661087 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:06.660919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-frrn9" event={"ID":"282366ea-d550-407f-a397-81dafba91392","Type":"ContainerStarted","Data":"91ae07593665cb3d02920e66c02e892ec867f873d9a1bb95f5cf401ca29bab6c"}
Jul 17 18:38:06.692248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:06.692190    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-frrn9" podStartSLOduration=3.4960444920000002 podStartE2EDuration="4.692174189s" totalImagesPullingTime="1.12217209s" totalInitContainerRuntime="73.957607ms" isStatefulPod=true podCreationTimestamp="2026-07-17 18:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:06.690689167 +0000 UTC m=+295.601515930" watchObservedRunningTime="2026-07-17 18:38:06.692174189 +0000 UTC m=+295.603000950"
Jul 17 18:38:07.690645 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:38:07.690595    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hmfvz" podUID="e0553119-5c66-4b8e-9e4b-c68c8f06af08"
Jul 17 18:38:07.723563 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:07.723527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:07.795929 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:07.795893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:08.595715 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.595689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b7d76964d-pht9r"
Jul 17 18:38:08.860733 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.860645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:08.861210 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.861040    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:38:08.867050 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.867018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.870354 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.870326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 18:38:08.870529 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.870512    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 18:38:08.870594 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.870536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lt4rp\""
Jul 17 18:38:08.870805 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.870783    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 18:38:08.870903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.870816    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 18:38:08.870903 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.870822    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 18:38:08.871096 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.871079    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 18:38:08.871308 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.871292    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-o2ps1q2gkr8i\""
Jul 17 18:38:08.871511 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.871494    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 18:38:08.871616 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.871532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 18:38:08.877882 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.877864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 18:38:08.878306 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.878289    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 18:38:08.878395 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.878377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 18:38:08.879060 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.879044    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 18:38:08.879386 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.879372    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 18:38:08.881131 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.881094    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 18:38:08.886838 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.886819    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 18:38:08.913207 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.913171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:38:08.995240 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995421 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/016bbbfb-9726-42d2-9ad9-6485660e5edd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995421 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995421 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995421 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995421 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995421 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5dw\" (UniqueName: \"kubernetes.io/projected/016bbbfb-9726-42d2-9ad9-6485660e5edd-kube-api-access-jj5dw\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/016bbbfb-9726-42d2-9ad9-6485660e5edd-config-out\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995632 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-web-config\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995861 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995861 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-config\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995861 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995861 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:08.995861 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:08.995698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.025608 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.025580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:09.096328 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096328 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/016bbbfb-9726-42d2-9ad9-6485660e5edd-config-out\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-web-config\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-config\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096523 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/016bbbfb-9726-42d2-9ad9-6485660e5edd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5dw\" (UniqueName: \"kubernetes.io/projected/016bbbfb-9726-42d2-9ad9-6485660e5edd-kube-api-access-jj5dw\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.096863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.097530 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.096851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.097530 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.097210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.097530 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.097335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.097690 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.097561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.099095 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.098556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.099337 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.099313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.100064 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.100024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/016bbbfb-9726-42d2-9ad9-6485660e5edd-config-out\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.100189 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.100146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.100333 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.100310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.101738 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.101687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-config\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.101996 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.101864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016bbbfb-9726-42d2-9ad9-6485660e5edd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.102137 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.102096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.102321 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.102295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-web-config\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.102541 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.102524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.102620 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.102599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.102792 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.102774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.102879 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.102858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.103217 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.103197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/016bbbfb-9726-42d2-9ad9-6485660e5edd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.103519 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.103504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/016bbbfb-9726-42d2-9ad9-6485660e5edd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.113102 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.113043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5dw\" (UniqueName: \"kubernetes.io/projected/016bbbfb-9726-42d2-9ad9-6485660e5edd-kube-api-access-jj5dw\") pod \"prometheus-k8s-0\" (UID: \"016bbbfb-9726-42d2-9ad9-6485660e5edd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.123860 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.123830    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" podUID="7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 17 18:38:09.176699 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.176663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:09.193662 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.193618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:09.354428 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.354391    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:38:09.359150 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:38:09.359107    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016bbbfb_9726_42d2_9ad9_6485660e5edd.slice/crio-674ca69fab63b20d3fc04ceeea1cec8bcc3f6d4c40661a7b8c3d471fdac909c6 WatchSource:0}: Error finding container 674ca69fab63b20d3fc04ceeea1cec8bcc3f6d4c40661a7b8c3d471fdac909c6: Status 404 returned error can't find the container with id 674ca69fab63b20d3fc04ceeea1cec8bcc3f6d4c40661a7b8c3d471fdac909c6
Jul 17 18:38:09.637546 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.637451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7dccw"
Jul 17 18:38:09.672015 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:09.671977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"016bbbfb-9726-42d2-9ad9-6485660e5edd","Type":"ContainerStarted","Data":"674ca69fab63b20d3fc04ceeea1cec8bcc3f6d4c40661a7b8c3d471fdac909c6"}
Jul 17 18:38:11.129871 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.129834    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl" podUID="da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" containerName="registry" containerID="cri-o://ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c" gracePeriod=30
Jul 17 18:38:11.381083 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.381059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:38:11.517425 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") pod \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") "
Jul 17 18:38:11.517425 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:38:11.517652 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lmzm\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-kube-api-access-7lmzm\") pod \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") "
Jul 17 18:38:11.517652 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-ca-trust-extracted\") pod \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") "
Jul 17 18:38:11.517652 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-installation-pull-secrets\") pod \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") "
Jul 17 18:38:11.517652 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517597    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-image-registry-private-configuration\") pod \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") "
Jul 17 18:38:11.517652 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517651    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-bound-sa-token\") pod \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") "
Jul 17 18:38:11.517893 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-certificates\") pod \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") "
Jul 17 18:38:11.517893 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.517722    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-trusted-ca\") pod \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\" (UID: \"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6\") "
Jul 17 18:38:11.518404 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.518370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:11.518562 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.518535    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:11.520474 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.520245    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-kube-api-access-7lmzm" (OuterVolumeSpecName: "kube-api-access-7lmzm") pod "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6"). InnerVolumeSpecName "kube-api-access-7lmzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:38:11.520474 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.520339    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:38:11.520474 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.520425    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:38:11.521249 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.521221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:38:11.521680 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.521649    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:38:11.522174 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.522154    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:38:11.523689 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.523671    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 18:38:11.528709 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.528683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" (UID: "da3948d5-fbd0-43ed-8e2e-bf1961e94fb6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:38:11.618985 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.618949    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 18:38:11.618985 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.618976    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lmzm\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-kube-api-access-7lmzm\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 18:38:11.618985 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.618985    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-ca-trust-extracted\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 18:38:11.618985 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.618995    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-installation-pull-secrets\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 18:38:11.619283 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.619005    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-image-registry-private-configuration\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 18:38:11.619283 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.619016    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-bound-sa-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 18:38:11.619283 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.619024    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-registry-certificates\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 18:38:11.619283 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.619032    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6-trusted-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 18:38:11.679439 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.679272    2563 generic.go:356] "Generic (PLEG): container finished" podID="016bbbfb-9726-42d2-9ad9-6485660e5edd" containerID="7d002b9689ab58706c2e1ed6c155bd1832db78488dedbfec389f4c16c876e556" exitCode=0
Jul 17 18:38:11.679439 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.679364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"016bbbfb-9726-42d2-9ad9-6485660e5edd","Type":"ContainerDied","Data":"7d002b9689ab58706c2e1ed6c155bd1832db78488dedbfec389f4c16c876e556"}
Jul 17 18:38:11.696394 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.680568    2563 generic.go:356] "Generic (PLEG): container finished" podID="da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" containerID="ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c" exitCode=0
Jul 17 18:38:11.696394 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.680605    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl"
Jul 17 18:38:11.696394 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.680627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl" event={"ID":"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6","Type":"ContainerDied","Data":"ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c"}
Jul 17 18:38:11.696394 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.680658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6564dc85c9-dgznl" event={"ID":"da3948d5-fbd0-43ed-8e2e-bf1961e94fb6","Type":"ContainerDied","Data":"49b9d9b06a77ba7b1a755180800c4b6d8f3cdca4c8adf020ed834b4d4d528155"}
Jul 17 18:38:11.696394 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.680677    2563 scope.go:117] "RemoveContainer" containerID="ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c"
Jul 17 18:38:11.696394 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.689199    2563 scope.go:117] "RemoveContainer" containerID="ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c"
Jul 17 18:38:11.696394 ip-10-0-138-130 kubenswrapper[2563]: E0717 18:38:11.689482    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c\": container with ID starting with ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c not found: ID does not exist" containerID="ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c"
Jul 17 18:38:11.696394 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.689507    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c"} err="failed to get container status \"ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c\": rpc error: code = NotFound desc = could not find container \"ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c\": container with ID starting with ff708027b6424f96e2b02455614d65e1e88e90a425a794626d20581af961894c not found: ID does not exist"
Jul 17 18:38:11.774324 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.774231    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6564dc85c9-dgznl"]
Jul 17 18:38:11.779962 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.779934    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6564dc85c9-dgznl"]
Jul 17 18:38:11.798704 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.798640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:11.941178 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:11.941141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:12.276584 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:12.276547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:13.054089 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:13.054039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:13.157147 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:13.157085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:13.693079 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:13.693045    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" path="/var/lib/kubelet/pods/da3948d5-fbd0-43ed-8e2e-bf1961e94fb6/volumes"
Jul 17 18:38:16.701064 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:16.701036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"016bbbfb-9726-42d2-9ad9-6485660e5edd","Type":"ContainerStarted","Data":"3ca1bbfd94e40fce5670423eff7d6eefafc82d4747679cb5cf7b62411f52d181"}
Jul 17 18:38:16.781764 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:16.781742    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:38:17.708169 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:17.708104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"016bbbfb-9726-42d2-9ad9-6485660e5edd","Type":"ContainerStarted","Data":"6bd76f89d8ffa754dd3d72e56b26dddb39faa1856e6cffcd1cd94d4aa3034634"}
Jul 17 18:38:18.856807 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:18.856769    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:18.960212 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:18.960173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:19.058901 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:19.058859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:19.123777 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:19.123689    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" podUID="7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 17 18:38:19.155643 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:19.155604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:19.689288 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:19.689239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:38:19.719459 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:19.719423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"016bbbfb-9726-42d2-9ad9-6485660e5edd","Type":"ContainerStarted","Data":"0e4433021cf1f12d5038f8101b57f4984934d929193e1221b8718de404209c57"}
Jul 17 18:38:20.725726 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:20.725652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:20.726857 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:20.726834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"016bbbfb-9726-42d2-9ad9-6485660e5edd","Type":"ContainerStarted","Data":"6deb38cb18d96010783e06ad3c4fd6642aa25770c1a25b8131b5144c9893094e"}
Jul 17 18:38:20.726931 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:20.726892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"016bbbfb-9726-42d2-9ad9-6485660e5edd","Type":"ContainerStarted","Data":"94d803e93a75b706c1f1db6dcde6ec5408cbdcfb7c181d5b4b2c981e766a1d62"}
Jul 17 18:38:20.726931 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:20.726919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"016bbbfb-9726-42d2-9ad9-6485660e5edd","Type":"ContainerStarted","Data":"22043da1d852a65c5e4470e07a743a80628ed72a21dda8dfa72da288c9043e0c"}
Jul 17 18:38:20.765502 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:20.765446    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.592076594 podStartE2EDuration="12.76542722s" totalImagesPullingTime="9.154243239s" totalInitContainerRuntime="19.107387ms" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:20.763498449 +0000 UTC m=+309.674325221" watchObservedRunningTime="2026-07-17 18:38:20.76542722 +0000 UTC m=+309.676253981"
Jul 17 18:38:20.825741 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:20.825706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:24.177098 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:24.177053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:38:28.099494 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:28.099458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:28.198067 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:28.198020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:29.124255 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:29.124216    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" podUID="7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 17 18:38:29.124726 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:29.124281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg"
Jul 17 18:38:29.124783 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:29.124734    2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b66709576e9099d7fc5fe8b07fc7f4c3c6dc31ec2704434502528dc631f04262"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 17 18:38:29.124783 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:29.124769    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" podUID="7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e" containerName="service-proxy" containerID="cri-o://b66709576e9099d7fc5fe8b07fc7f4c3c6dc31ec2704434502528dc631f04262" gracePeriod=30
Jul 17 18:38:29.759470 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:29.759429    2563 generic.go:356] "Generic (PLEG): container finished" podID="7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e" containerID="b66709576e9099d7fc5fe8b07fc7f4c3c6dc31ec2704434502528dc631f04262" exitCode=2
Jul 17 18:38:29.759635 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:29.759481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" event={"ID":"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e","Type":"ContainerDied","Data":"b66709576e9099d7fc5fe8b07fc7f4c3c6dc31ec2704434502528dc631f04262"}
Jul 17 18:38:29.759635 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:29.759535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65554f86c6-xwzvg" event={"ID":"7809eb55-cc3b-4b9a-9dbb-c59f4827ea6e","Type":"ContainerStarted","Data":"f297507286ef9f0565b3124157eb18a8d3a54fac5a0c6e12c0ad5919ef583fee"}
Jul 17 18:38:31.828218 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:31.828191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tqqs_df37e276-43e7-41eb-88e6-4f78efcbcd7d/serve-healthcheck-canary/0.log"
Jul 17 18:38:32.996059 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:32.996023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:33.069389 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:33.069342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:40.680930 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:40.680886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:40.919369 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:40.919337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:41.310505 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:41.310466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:47.987857 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:47.987803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:38:48.087227 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:38:48.087191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:09.177866 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:09.177761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:39:09.197997 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:09.197972    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:39:09.910228 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:09.910201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:39:20.074091 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:20.074049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:20.163458 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:20.163425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:21.823004 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:21.822971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:22.172183 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:22.172150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:25.532231 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:25.532187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:39:25.534732 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:25.534709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0553119-5c66-4b8e-9e4b-c68c8f06af08-metrics-certs\") pod \"network-metrics-daemon-hmfvz\" (UID: \"e0553119-5c66-4b8e-9e4b-c68c8f06af08\") " pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:39:25.664462 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:25.664421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:25.692449 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:25.692421    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gbnj9\""
Jul 17 18:39:25.700490 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:25.700465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmfvz"
Jul 17 18:39:25.752761 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:25.752726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:25.844001 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:25.843972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hmfvz"]
Jul 17 18:39:25.847411 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:39:25.847385    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0553119_5c66_4b8e_9e4b_c68c8f06af08.slice/crio-0dc9280d0bd95a0c92a83423444fe18df3db96168670030b9ae4d0e8165e9f58 WatchSource:0}: Error finding container 0dc9280d0bd95a0c92a83423444fe18df3db96168670030b9ae4d0e8165e9f58: Status 404 returned error can't find the container with id 0dc9280d0bd95a0c92a83423444fe18df3db96168670030b9ae4d0e8165e9f58
Jul 17 18:39:25.948357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:25.948320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hmfvz" event={"ID":"e0553119-5c66-4b8e-9e4b-c68c8f06af08","Type":"ContainerStarted","Data":"0dc9280d0bd95a0c92a83423444fe18df3db96168670030b9ae4d0e8165e9f58"}
Jul 17 18:39:26.953996 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:26.953961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hmfvz" event={"ID":"e0553119-5c66-4b8e-9e4b-c68c8f06af08","Type":"ContainerStarted","Data":"f8d82600d85d8d591f7cbcef7220464e9fe541c4e3be8b4cc2bc82f736645863"}
Jul 17 18:39:26.954390 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:26.954011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hmfvz" event={"ID":"e0553119-5c66-4b8e-9e4b-c68c8f06af08","Type":"ContainerStarted","Data":"66019cc1709ed0d35fbbb2b4f1527bd34677eea01f940ec88da0cfd676db8b48"}
Jul 17 18:39:26.980732 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:26.980629    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hmfvz" podStartSLOduration=375.108727576 podStartE2EDuration="6m15.980608659s" totalImagesPullingTime="871.881083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:39:26.979703706 +0000 UTC m=+375.890530480" watchObservedRunningTime="2026-07-17 18:39:26.980608659 +0000 UTC m=+375.891435421"
Jul 17 18:39:27.165218 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:27.165162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:27.242332 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:27.242236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:31.556092 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:31.556058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:31.646723 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:31.646675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:37.237932 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:37.237896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:37.300561 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:37.300501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:49.198272 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:49.198236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:49.298130 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:49.298064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:50.278724 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:50.278673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:50.394216 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:50.394181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:39:50.528918 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:39:50.528828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:40:02.018905 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:40:02.018869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:40:03.019106 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:40:03.019067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:40:03.115525 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:40:03.115488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:40:41.780820 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:40:41.780731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:40:41.871009 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:40:41.870972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:09.088898 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:09.088865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:10.295392 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:10.295347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:13.079829 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:13.079787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:13.185887 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:13.185822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:16.036436 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:16.036393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:16.149965 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:16.149934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:17.559302 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:17.559258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:17.782357 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:17.782315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:19.879140 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:19.879082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:19.978858 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:19.978822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:25.183676 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:25.183640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:26.993536 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:26.993495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:27.096292 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:27.096259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:27.192151 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:27.192103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:30.311318 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:30.311284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:30.407611 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:30.407571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:33.201166 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:33.201107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:33.307819 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:33.307777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:35.350614 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:35.350581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:47.380394 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:47.380361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:47.509691 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:47.509654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:47.744612 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:47.744524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:48.113762 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.113725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:48.202248 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.202202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:48.277067 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.277030    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"]
Jul 17 18:41:48.277375 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.277358    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" containerName="registry"
Jul 17 18:41:48.277458 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.277377    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" containerName="registry"
Jul 17 18:41:48.277458 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.277440    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3948d5-fbd0-43ed-8e2e-bf1961e94fb6" containerName="registry"
Jul 17 18:41:48.280710 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.280688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.293842 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.293817    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 18:41:48.294276 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.294260    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n2nnj\""
Jul 17 18:41:48.294767 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.294754    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 18:41:48.295274 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.295261    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 18:41:48.295321 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.295298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 18:41:48.344227 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.344192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"]
Jul 17 18:41:48.349968 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.349943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.350082 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.350000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.350150 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.350077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.350189 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.350149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.350223 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.350186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj56s\" (UniqueName: \"kubernetes.io/projected/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-kube-api-access-hj56s\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.451441 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.451349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.451441 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.451396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.451854 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.451522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.451854 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.451577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj56s\" (UniqueName: \"kubernetes.io/projected/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-kube-api-access-hj56s\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.451854 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.451618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.451962 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.451904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.454080 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.454049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.454231 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.454207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.454280 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.454263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.466737 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.466706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj56s\" (UniqueName: \"kubernetes.io/projected/9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d-kube-api-access-hj56s\") pod \"opendatahub-operator-controller-manager-57479f7bfb-bjjdw\" (UID: \"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.485832 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.485795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:48.584823 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.584785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:48.590693 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.590661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:48.684076 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.683953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:48.734424 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.734278    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"]
Jul 17 18:41:48.739495 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:41:48.739466    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc7b91d_6fdd_44fc_8ac0_d1594aec5e5d.slice/crio-6973168cc01b670280fa3264961187158baeb489e8d05c0df70927aafe397911 WatchSource:0}: Error finding container 6973168cc01b670280fa3264961187158baeb489e8d05c0df70927aafe397911: Status 404 returned error can't find the container with id 6973168cc01b670280fa3264961187158baeb489e8d05c0df70927aafe397911
Jul 17 18:41:48.789264 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.789233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:48.893132 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.893078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:48.989561 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:48.989471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:49.438808 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:49.438749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw" event={"ID":"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d","Type":"ContainerStarted","Data":"6973168cc01b670280fa3264961187158baeb489e8d05c0df70927aafe397911"}
Jul 17 18:41:52.451597 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:52.451562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw" event={"ID":"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d","Type":"ContainerStarted","Data":"1d49ffd9d6e5cd5d22483280826ca3d18bcea7011ca194e227b61b2d8950a6ca"}
Jul 17 18:41:53.390796 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:53.390764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:53.457242 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:53.457214    2563 generic.go:356] "Generic (PLEG): container finished" podID="9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d" containerID="1d49ffd9d6e5cd5d22483280826ca3d18bcea7011ca194e227b61b2d8950a6ca" exitCode=0
Jul 17 18:41:53.457605 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:53.457277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw" event={"ID":"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d","Type":"ContainerDied","Data":"1d49ffd9d6e5cd5d22483280826ca3d18bcea7011ca194e227b61b2d8950a6ca"}
Jul 17 18:41:53.768308 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:53.768224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:53.887469 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:53.887433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:41:54.462717 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:54.462679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw" event={"ID":"9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d","Type":"ContainerStarted","Data":"e4472b8d03f35eb9fd9ce6b4efe3080785697829397c754d1c6caa652cdc4f38"}
Jul 17 18:41:54.463091 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:54.462797    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:41:54.489660 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:41:54.489604    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw" podStartSLOduration=1.941133051 podStartE2EDuration="6.489582995s" totalImagesPullingTime="3.463412614s" totalInitContainerRuntime="1.08503733s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:41:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:41:54.488182365 +0000 UTC m=+523.399009154" watchObservedRunningTime="2026-07-17 18:41:54.489582995 +0000 UTC m=+523.400409757"
Jul 17 18:42:05.467734 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:05.467646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-bjjdw"
Jul 17 18:42:07.432341 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:07.432307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:07.586333 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:07.586293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:08.270485 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.270450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:08.413164 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.413113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:08.484113 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.484077    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"]
Jul 17 18:42:08.490154 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.490108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.493197 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.493176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mkcfh\""
Jul 17 18:42:08.493320 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.493227    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 18:42:08.494207 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.494189    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:42:08.494207 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.494201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 18:42:08.494371 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.494227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 18:42:08.494371 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.494316    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 18:42:08.499074 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.499051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"]
Jul 17 18:42:08.537789 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.537706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:08.610971 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.610943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhw4\" (UniqueName: \"kubernetes.io/projected/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-kube-api-access-bmhw4\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.611162 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.610991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-metrics-cert\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.611162 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.611035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-manager-config\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.611162 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.611051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-cert\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.629289 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.629251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:08.711722 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.711675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-manager-config\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.711722 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.711725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-cert\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.711937 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.711787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhw4\" (UniqueName: \"kubernetes.io/projected/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-kube-api-access-bmhw4\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.712009 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.711989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-metrics-cert\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.712539 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.712441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-manager-config\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.714443 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.714423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-metrics-cert\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.714527 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.714461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-cert\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.723562 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.723544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhw4\" (UniqueName: \"kubernetes.io/projected/c2b38ddf-00a1-4c6a-a20b-5f3b480c7055-kube-api-access-bmhw4\") pod \"lws-controller-manager-cbc4544f6-dl9ct\" (UID: \"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.760779 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.760751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:08.799809 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.799729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:08.939691 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:08.939662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"]
Jul 17 18:42:08.942469 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:42:08.942439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b38ddf_00a1_4c6a_a20b_5f3b480c7055.slice/crio-4d6a9ee3952df112583f1382c5656c6822d3bafe0b3890dcd8ea79a2a12dd010 WatchSource:0}: Error finding container 4d6a9ee3952df112583f1382c5656c6822d3bafe0b3890dcd8ea79a2a12dd010: Status 404 returned error can't find the container with id 4d6a9ee3952df112583f1382c5656c6822d3bafe0b3890dcd8ea79a2a12dd010
Jul 17 18:42:09.227890 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:09.227840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:09.505969 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:09.505881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:09.516918 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:09.516879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct" event={"ID":"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055","Type":"ContainerStarted","Data":"4d6a9ee3952df112583f1382c5656c6822d3bafe0b3890dcd8ea79a2a12dd010"}
Jul 17 18:42:10.890048 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:10.890006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:11.014715 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:11.014664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:11.525459 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:11.525373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct" event={"ID":"c2b38ddf-00a1-4c6a-a20b-5f3b480c7055","Type":"ContainerStarted","Data":"acaa8427529a8f9f3502bc5d3d2c2b1387280e7a8271309774853f45dc26335b"}
Jul 17 18:42:11.525623 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:11.525513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:11.552237 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:11.552195    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct" podStartSLOduration=1.342271168 podStartE2EDuration="3.552180487s" totalImagesPullingTime="2.209909319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:11.552052025 +0000 UTC m=+540.462878788" watchObservedRunningTime="2026-07-17 18:42:11.552180487 +0000 UTC m=+540.463007248"
Jul 17 18:42:12.471599 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:12.471564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:16.402245 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.402208    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"]
Jul 17 18:42:16.405728 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.405706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.408819 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.408798    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 18:42:16.409139 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.409100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 18:42:16.409214 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.409148    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 18:42:16.409214 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.409201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 18:42:16.409843 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.409827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 18:42:16.410201 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.410186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d5ww9\""
Jul 17 18:42:16.410630 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.410617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 18:42:16.427517 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.427484    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"]
Jul 17 18:42:16.482044 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.482001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b227c4dd-4337-402e-a5e6-5fa1d383a6da-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.482044 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.482048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.482288 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.482079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.482288 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.482097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.482288 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.482136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjzm\" (UniqueName: \"kubernetes.io/projected/b227c4dd-4337-402e-a5e6-5fa1d383a6da-kube-api-access-jkjzm\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.482288 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.482217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.482288 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.482256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.582845 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.582812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.583045 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.582851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.583045 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.582878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b227c4dd-4337-402e-a5e6-5fa1d383a6da-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.583045 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.582901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.583045 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.582938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.583045 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.582957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.583045 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.582979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjzm\" (UniqueName: \"kubernetes.io/projected/b227c4dd-4337-402e-a5e6-5fa1d383a6da-kube-api-access-jkjzm\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.583859 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.583706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.585413 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.585383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b227c4dd-4337-402e-a5e6-5fa1d383a6da-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.585413 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.585392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.585571 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.585448    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.585571 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.585505    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b227c4dd-4337-402e-a5e6-5fa1d383a6da-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.609066 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.609031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjzm\" (UniqueName: \"kubernetes.io/projected/b227c4dd-4337-402e-a5e6-5fa1d383a6da-kube-api-access-jkjzm\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.610023 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.610003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b227c4dd-4337-402e-a5e6-5fa1d383a6da-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dbvjc\" (UID: \"b227c4dd-4337-402e-a5e6-5fa1d383a6da\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.715344 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.715248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:16.863737 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.863704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"]
Jul 17 18:42:16.873715 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:42:16.873681    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb227c4dd_4337_402e_a5e6_5fa1d383a6da.slice/crio-00a002bd15e6a32c73982e2cc82bcf6edc98578c68308746d2855d289b9c4533 WatchSource:0}: Error finding container 00a002bd15e6a32c73982e2cc82bcf6edc98578c68308746d2855d289b9c4533: Status 404 returned error can't find the container with id 00a002bd15e6a32c73982e2cc82bcf6edc98578c68308746d2855d289b9c4533
Jul 17 18:42:16.891393 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:16.891367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:17.024401 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:17.024302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:17.549568 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:17.549522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc" event={"ID":"b227c4dd-4337-402e-a5e6-5fa1d383a6da","Type":"ContainerStarted","Data":"00a002bd15e6a32c73982e2cc82bcf6edc98578c68308746d2855d289b9c4533"}
Jul 17 18:42:17.657848 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:17.657800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:17.953692 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:17.953646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:19.366479 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:19.366449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:19.953096 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:19.953059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:20.082051 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.082001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:20.231780 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.231744    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 18:42:20.231859 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.231809    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 18:42:20.384235 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.384194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:20.563993 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.563895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc" event={"ID":"b227c4dd-4337-402e-a5e6-5fa1d383a6da","Type":"ContainerStarted","Data":"8619dcb741d8d23e700ff7eabb4f92078f9f63ce07dd5efa7ffcf87c0071a020"}
Jul 17 18:42:20.564184 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.564018    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:20.650108 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.650033    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc" podStartSLOduration=1.294106577 podStartE2EDuration="4.650009836s" totalImagesPullingTime="3.355903259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:20.647489644 +0000 UTC m=+549.558316416" watchObservedRunningTime="2026-07-17 18:42:20.650009836 +0000 UTC m=+549.560836598"
Jul 17 18:42:20.737731 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.737687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:20.910098 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:20.910063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:21.569582 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:21.569546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dbvjc"
Jul 17 18:42:22.531183 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:22.531156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-dl9ct"
Jul 17 18:42:44.217075 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:44.217040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:44.291906 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:44.291865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:42:44.941676 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:42:44.941629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:08.755741 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:08.755699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:09.629590 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:09.629549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:09.706895 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:09.706851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:11.542895 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:11.542863    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:43:11.542895 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:11.542894    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:43:32.319014 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:32.318931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:36.314861 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:36.314825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:38.979928 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:38.979890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:39.363984 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:39.363952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:39.501870 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:39.501841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:42.906762 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:42.906728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:44.170604 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:44.170571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:44.279339 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:44.279294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:46.184689 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:46.184649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:47.983322 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:47.983282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:49.348343 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:49.348307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:49.440080 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:49.440052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:53.125834 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:53.125801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:53.226329 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:53.226293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:56.361684 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:56.361645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:56.434069 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:56.434033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:57.314837 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:57.314801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:57.418289 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:57.418259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:57.764277 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:57.764242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:43:57.865933 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:43:57.865886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:00.745492 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:00.745455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:00.848199 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:00.848162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:01.232435 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:01.232401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:01.331708 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:01.331674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:07.410840 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:07.410795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:07.576392 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:07.576351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:10.259493 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:10.259442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:10.366715 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:10.366683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:15.078432 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:15.078395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:15.171637 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:15.171571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:15.288752 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:15.288717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:18.788186 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:18.788143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:18.886424 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:18.886374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:25.378736 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:25.378654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:25.484112 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:25.484076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:27.330990 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:27.330955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:29.173585 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:29.173545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:29.272188 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:29.272148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:33.568908 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:33.568876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:33.670298 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:33.670257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:33.976481 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:33.976448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:34.076404 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:34.076371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:35.905617 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:35.905570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:36.547208 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:36.547170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:39.168723 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:39.168683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:39.242829 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:39.242791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:39.346151 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:39.346086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:39.447818 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:39.447713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:40.092390 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:40.092358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:40.184281 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:40.184254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:50.856381 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:50.856343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:50.959731 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:50.959700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:51.024916 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.024878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:51.285777 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.285693    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:44:51.288748 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.288731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:51.291134 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.291083    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 18:44:51.291265 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.291138    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tzbmh\""
Jul 17 18:44:51.292184 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.292169    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 18:44:51.292268 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.292188    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 18:44:51.298672 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.298650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:44:51.333319 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.333285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:44:51.340841 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.340799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:51.342834 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.342814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/87518cbb-e489-4491-852f-819eca0aa98b-config-file\") pod \"limitador-limitador-54d6d988d5-9ctsp\" (UID: \"87518cbb-e489-4491-852f-819eca0aa98b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:51.342937 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.342857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8jkb\" (UniqueName: \"kubernetes.io/projected/87518cbb-e489-4491-852f-819eca0aa98b-kube-api-access-p8jkb\") pod \"limitador-limitador-54d6d988d5-9ctsp\" (UID: \"87518cbb-e489-4491-852f-819eca0aa98b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:51.438782 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.438743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:51.443799 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.443776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/87518cbb-e489-4491-852f-819eca0aa98b-config-file\") pod \"limitador-limitador-54d6d988d5-9ctsp\" (UID: \"87518cbb-e489-4491-852f-819eca0aa98b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:51.443909 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.443820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8jkb\" (UniqueName: \"kubernetes.io/projected/87518cbb-e489-4491-852f-819eca0aa98b-kube-api-access-p8jkb\") pod \"limitador-limitador-54d6d988d5-9ctsp\" (UID: \"87518cbb-e489-4491-852f-819eca0aa98b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:51.444407 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.444388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/87518cbb-e489-4491-852f-819eca0aa98b-config-file\") pod \"limitador-limitador-54d6d988d5-9ctsp\" (UID: \"87518cbb-e489-4491-852f-819eca0aa98b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:51.452888 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.452869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8jkb\" (UniqueName: \"kubernetes.io/projected/87518cbb-e489-4491-852f-819eca0aa98b-kube-api-access-p8jkb\") pod \"limitador-limitador-54d6d988d5-9ctsp\" (UID: \"87518cbb-e489-4491-852f-819eca0aa98b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:51.599050 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.599018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:51.722954 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.722921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:51.736527 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.736430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:44:51.739082 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:44:51.739057    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87518cbb_e489_4491_852f_819eca0aa98b.slice/crio-1c742123ff7898d1e0ffcd4d33f7e8d1b14d7288bcf8d8b05970adcc71e06012 WatchSource:0}: Error finding container 1c742123ff7898d1e0ffcd4d33f7e8d1b14d7288bcf8d8b05970adcc71e06012: Status 404 returned error can't find the container with id 1c742123ff7898d1e0ffcd4d33f7e8d1b14d7288bcf8d8b05970adcc71e06012
Jul 17 18:44:51.740851 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.740835    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:44:51.817356 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:51.817318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:44:52.096997 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:52.096969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp" event={"ID":"87518cbb-e489-4491-852f-819eca0aa98b","Type":"ContainerStarted","Data":"1c742123ff7898d1e0ffcd4d33f7e8d1b14d7288bcf8d8b05970adcc71e06012"}
Jul 17 18:44:55.110484 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:55.110449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp" event={"ID":"87518cbb-e489-4491-852f-819eca0aa98b","Type":"ContainerStarted","Data":"eb9cd70edf8d27d1e73291da8b9ec5667811b82d5ee68c2a2f467d8fe375d40c"}
Jul 17 18:44:55.110964 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:55.110603    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:44:55.128963 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:44:55.128915    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp" podStartSLOduration=1.691389512 podStartE2EDuration="4.128900626s" totalImagesPullingTime="2.437511114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:55.12740798 +0000 UTC m=+704.038234753" watchObservedRunningTime="2026-07-17 18:44:55.128900626 +0000 UTC m=+704.039727415"
Jul 17 18:45:06.114991 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:06.114920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"
Jul 17 18:45:09.025724 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:09.025673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:09.120082 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:09.120044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:09.507064 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:09.507035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:21.598417 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:21.598376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:21.868685 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:21.868589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:28.429774 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:28.429727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:28.528504 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:28.528459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:28.587807 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:28.587758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:28.847138 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:28.847070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:31.739513 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:31.739474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:45:31.840346 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:45:31.840310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:03.831562 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:03.831523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:03.925863 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:03.925829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:04.470655 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:04.470618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:04.758026 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:04.757934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:05.008975 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:05.008885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:05.114219 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:05.114185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:05.198232 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:05.198193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:46:15.152785 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:15.152749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:15.249552 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:15.249511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:21.634650 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:21.634612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:22.114452 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:22.114420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:46:26.530779 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:26.530742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:27.505030 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:27.504991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:46:35.025245 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:35.025154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:36.309862 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:36.309821    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:46:44.826141 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:44.826081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:46:46.503175 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:46:46.503139    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:47:02.042758 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.042723    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"]
Jul 17 18:47:02.046260 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.046239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.049445 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.049422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 18:47:02.049656 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.049642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 18:47:02.052739 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.052616    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 17 18:47:02.053302 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.053287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-skht6\""
Jul 17 18:47:02.071957 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.071925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"]
Jul 17 18:47:02.110576 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.110538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf5cd24-999a-4bce-8f38-1424810ed746-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.110770 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.110594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjnch\" (UniqueName: \"kubernetes.io/projected/baf5cd24-999a-4bce-8f38-1424810ed746-kube-api-access-vjnch\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.110770 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.110621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.110770 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.110675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.110770 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.110712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.110770 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.110767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.129062 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.129033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:47:02.212055 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjnch\" (UniqueName: \"kubernetes.io/projected/baf5cd24-999a-4bce-8f38-1424810ed746-kube-api-access-vjnch\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.212272 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.212272 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.212272 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.212272 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.212272 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf5cd24-999a-4bce-8f38-1424810ed746-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.212535 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.212579 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.212633 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.212615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.214575 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.214546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf5cd24-999a-4bce-8f38-1424810ed746-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.214909 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.214889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf5cd24-999a-4bce-8f38-1424810ed746-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.224388 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.224363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjnch\" (UniqueName: \"kubernetes.io/projected/baf5cd24-999a-4bce-8f38-1424810ed746-kube-api-access-vjnch\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9pnpb\" (UID: \"baf5cd24-999a-4bce-8f38-1424810ed746\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.356586 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.356547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:02.492693 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.492669    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"]
Jul 17 18:47:02.496717 ip-10-0-138-130 kubenswrapper[2563]: W0717 18:47:02.496684    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf5cd24_999a_4bce_8f38_1424810ed746.slice/crio-cd9256ce41cb2729a26c21bb6962a06ca34c3de035481f9873904fc4c82d107e WatchSource:0}: Error finding container cd9256ce41cb2729a26c21bb6962a06ca34c3de035481f9873904fc4c82d107e: Status 404 returned error can't find the container with id cd9256ce41cb2729a26c21bb6962a06ca34c3de035481f9873904fc4c82d107e
Jul 17 18:47:02.568644 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:02.568601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb" event={"ID":"baf5cd24-999a-4bce-8f38-1424810ed746","Type":"ContainerStarted","Data":"cd9256ce41cb2729a26c21bb6962a06ca34c3de035481f9873904fc4c82d107e"}
Jul 17 18:47:04.333354 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:04.333321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:47:04.576887 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:04.576852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb" event={"ID":"baf5cd24-999a-4bce-8f38-1424810ed746","Type":"ContainerStarted","Data":"2e4849f79f24a02b0ef2c4a616646a636a807c5c3fb4c1c6157aac59044c50e7"}
Jul 17 18:47:04.607305 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:04.607247    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb" podStartSLOduration=1.538430012 podStartE2EDuration="2.60723266s" totalImagesPullingTime="1.068802648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:04.606470957 +0000 UTC m=+833.517297717" watchObservedRunningTime="2026-07-17 18:47:04.60723266 +0000 UTC m=+833.518059420"
Jul 17 18:47:10.366495 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:10.366461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:47:12.356943 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:12.356899    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:12.357360 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:12.356961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:12.369550 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:12.369524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:47:12.607841 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:12.607755    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:47:12.618520 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:47:12.618501    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9pnpb"
Jul 17 18:48:10.095653 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:10.095553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:48:11.564830 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:11.564802    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:48:11.565323 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:11.565230    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:48:11.799615 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:11.799582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:48:39.858793 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:39.858760    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:48:49.517649 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:49.517613    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:48:59.622654 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:59.622620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:48:59.627184 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:59.627157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:48:59.723989 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:48:59.723935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:49:08.347572 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:49:08.347523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:49:29.738774 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:49:29.738732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:49:42.420451 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:49:42.420342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:49:42.880017 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:49:42.879981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:02.060697 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:02.060652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:15.345625 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:15.345592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:16.607232 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:16.607192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:41.207222 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:41.207187    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:50:50.408754 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:50.408718    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:50:50.988675 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:50.988634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:51.089831 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:51.089793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:51.207258 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:51.207229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:57.094481 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:57.094442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:57.204473 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:57.204435    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:57.293268 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:57.293230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:50:57.460283 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:50:57.460187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:51:03.452881 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:51:03.452771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:51:03.553819 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:51:03.553786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:51:21.009868 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:51:21.009831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:51:30.608831 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:51:30.608793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:51:41.309342 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:51:41.309304    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:51:48.804025 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:51:48.803990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:51:58.415452 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:51:58.415410    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:52:10.907328 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:52:10.907293    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:52:21.813407 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:52:21.813365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:52:32.106346 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:52:32.106251    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:52:53.314401 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:52:53.314366    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:53:05.409019 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:53:05.408985    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:53:11.586772 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:53:11.586738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:53:11.587540 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:53:11.587507    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:53:29.410681 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:53:29.410640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:53:40.616957 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:53:40.616915    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:53:58.304700 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:53:58.304660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:54:09.414041 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:54:09.413958    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:54:20.018717 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:54:20.018679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:54:32.800018 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:54:32.799983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:55:11.973607 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:55:11.973567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:56:24.503558 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:56:24.503521    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:56:37.012801 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:56:37.012764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:56:46.603623 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:56:46.603584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:57:00.310718 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:57:00.310684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:57:10.503941 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:57:10.503858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:57:11.857166 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:57:11.857111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:57:20.806871 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:57:20.806834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:57:28.806567 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:57:28.806529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:57:31.876280 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:57:31.876244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 18:57:40.113912 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:57:40.113876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:57:52.004566 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:57:52.004524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:58:00.728279 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:58:00.728244    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:58:11.608965 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:58:11.608935    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:58:11.610678 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:58:11.610657    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 18:58:14.832469 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:58:14.832435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:58:32.710519 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:58:32.710440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:58:42.317413 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:58:42.317376    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:58:52.438358 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:58:52.438321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:59:04.206217 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:59:04.206179    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:59:24.608358 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:59:24.608320    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:59:34.408329 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:59:34.408291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:59:44.116578 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:59:44.116541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 18:59:54.606922 ip-10-0-138-130 kubenswrapper[2563]: I0717 18:59:54.606885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:00:04.310561 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:00:04.310470    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:00:15.310701 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:00:15.310662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:00:33.701076 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:00:33.701042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:00:46.311727 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:00:46.311687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:00:59.208719 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:00:59.208683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:01:08.809953 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:01:08.809909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:01:19.583614 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:01:19.583573    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:01:32.924056 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:01:32.923977    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:01:48.710086 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:01:48.710044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:02:00.802217 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:00.802181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:02:03.674423 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:03.674384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:10.405157 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:10.405099    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:02:15.322241 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:15.322204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:23.588409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:23.588377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:28.694207 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:28.694173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:31.410362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:31.410329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:02:31.590232 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:31.590201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:31.712914 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:31.712831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:32.066095 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:32.066016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:38.570354 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.570318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:38.674300 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.674265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:38.760216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.760185    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"]
Jul 17 19:02:38.766686 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.766662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.769154 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.769111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7dc4fd53-openshift-default-dockercfg-hf2gt\""
Jul 17 19:02:38.790277 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.790251    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"]
Jul 17 19:02:38.795392 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.795366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:38.824225 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-socket\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.824225 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzjj\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-kube-api-access-hlzjj\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.824414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-data\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.824414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ab8c9-5117-43c6-be35-70cdd5033d0a-istiod-ca-cert\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.824414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-envoy\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.824414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-certs\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.824414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-token\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.824414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-credential-socket\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.824414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.824361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-podinfo\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.891818 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.891782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:38.925067 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-token\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925067 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-credential-socket\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-podinfo\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-socket\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzjj\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-kube-api-access-hlzjj\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-data\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ab8c9-5117-43c6-be35-70cdd5033d0a-istiod-ca-cert\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-envoy\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-certs\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-credential-socket\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-socket\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925747 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-certs\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.925878 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.925859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-data\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.926140 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.926102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ab8c9-5117-43c6-be35-70cdd5033d0a-istiod-ca-cert\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.927693 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.927670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-envoy\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.927855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.927839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-podinfo\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.934382 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.934343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-token\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:38.934484 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:38.934451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzjj\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-kube-api-access-hlzjj\") pod \"e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") " pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:39.076450 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:39.076359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:39.214201 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:39.214167    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"]
Jul 17 19:02:39.217827 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:02:39.217800    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918ab8c9_5117_43c6_be35_70cdd5033d0a.slice/crio-067a515802b4c904c59daeebbe3d4cebd0ff38414aa6bddc802c9a658289bc36 WatchSource:0}: Error finding container 067a515802b4c904c59daeebbe3d4cebd0ff38414aa6bddc802c9a658289bc36: Status 404 returned error can't find the container with id 067a515802b4c904c59daeebbe3d4cebd0ff38414aa6bddc802c9a658289bc36
Jul 17 19:02:39.219857 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:39.219839    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:02:40.047401 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:40.047362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" event={"ID":"918ab8c9-5117-43c6-be35-70cdd5033d0a","Type":"ContainerStarted","Data":"067a515802b4c904c59daeebbe3d4cebd0ff38414aa6bddc802c9a658289bc36"}
Jul 17 19:02:40.200896 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:40.200855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:40.365519 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:40.365479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:40.466417 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:40.466376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:40.657008 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:40.656866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:40.957736 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:40.957660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:41.909303 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:41.909262    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:02:41.909691 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:41.909354    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:02:41.909691 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:41.909402    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:02:42.059140 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:42.059078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" event={"ID":"918ab8c9-5117-43c6-be35-70cdd5033d0a","Type":"ContainerStarted","Data":"6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc"}
Jul 17 19:02:42.077364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:42.077328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:42.078977 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:42.078952    2563 patch_prober.go:28] interesting pod/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 17 19:02:42.079048 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:42.079025    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 17 19:02:42.084201 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:42.084148    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" podStartSLOduration=1.395205225 podStartE2EDuration="4.084109429s" totalImagesPullingTime="2.688904204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:02:42.08269452 +0000 UTC m=+1770.993521281" watchObservedRunningTime="2026-07-17 19:02:42.084109429 +0000 UTC m=+1770.994936190"
Jul 17 19:02:42.723907 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:42.723877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:43.081336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:43.081268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:44.070048 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:44.070016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:44.071147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:44.071107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:47.512675 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:47.512640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:02:53.606512 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:53.606464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:02:56.121977 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:56.121942    2563 generic.go:356] "Generic (PLEG): container finished" podID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerID="6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc" exitCode=0
Jul 17 19:02:56.122396 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:56.122004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" event={"ID":"918ab8c9-5117-43c6-be35-70cdd5033d0a","Type":"ContainerDied","Data":"6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc"}
Jul 17 19:02:56.122396 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:56.122384    2563 scope.go:117] "RemoveContainer" containerID="6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc"
Jul 17 19:02:56.123416 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:56.123379    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:02:56.123481 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:56.123464    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:02:56.123517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:56.123503    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:02:57.128128 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:57.128079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" event={"ID":"918ab8c9-5117-43c6-be35-70cdd5033d0a","Type":"ContainerStarted","Data":"51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444"}
Jul 17 19:02:58.077220 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:58.077181    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:58.077882 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:58.077845    2563 patch_prober.go:28] interesting pod/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 17 19:02:58.078021 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:58.077907    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 17 19:02:59.070588 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:59.070554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:02:59.077181 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:59.077143    2563 patch_prober.go:28] interesting pod/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 17 19:02:59.077362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:02:59.077218    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 17 19:03:00.077420 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:00.076961    2563 patch_prober.go:28] interesting pod/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 17 19:03:00.077420 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:00.077024    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 17 19:03:01.077833 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:01.077795    2563 patch_prober.go:28] interesting pod/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused" start-of-body=
Jul 17 19:03:01.078364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:01.077863    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.18:15021/healthz/ready\": dial tcp 10.134.0.18:15021: connect: connection refused"
Jul 17 19:03:02.081488 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:02.081407    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:03:02.082524 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:02.082502    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:03:11.632477 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:11.632438    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:03:11.633851 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:11.633830    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:03:51.936445 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:51.936407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:03:51.996683 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:03:51.996648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:14.710818 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:14.710782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:38.007243 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:38.007207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:38.107111 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:38.107073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:45.028195 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:45.028157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:45.051793 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:45.051754    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"]
Jul 17 19:05:45.051969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:45.051952    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy" containerID="cri-o://51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444" gracePeriod=30
Jul 17 19:05:45.126755 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:45.126717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:45.224998 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:45.224968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:47.084137 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:47.084086    2563 patch_prober.go:28] interesting pod/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 19:05:47.084515 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:47.084154    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 19:05:47.848395 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:47.848357    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:05:50.304973 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.304945    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:05:50.456431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456397    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ab8c9-5117-43c6-be35-70cdd5033d0a-istiod-ca-cert\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.456431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456448    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-podinfo\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.456689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-token\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.456689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlzjj\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-kube-api-access-hlzjj\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.456689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456541    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-envoy\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.456689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-credential-socket\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.456689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456603    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-data\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.456689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-certs\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.456689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-socket\") pod \"918ab8c9-5117-43c6-be35-70cdd5033d0a\" (UID: \"918ab8c9-5117-43c6-be35-70cdd5033d0a\") "
Jul 17 19:05:50.457036 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.456850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918ab8c9-5117-43c6-be35-70cdd5033d0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:05:50.457142 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.457095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:05:50.457209 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.457147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:05:50.457399 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.457321    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:05:50.457577 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.457531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:05:50.459093 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.459063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:05:50.459224 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.459086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-kube-api-access-hlzjj" (OuterVolumeSpecName: "kube-api-access-hlzjj") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "kube-api-access-hlzjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:05:50.459366 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.459347    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:05:50.459415 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.459370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "918ab8c9-5117-43c6-be35-70cdd5033d0a" (UID: "918ab8c9-5117-43c6-be35-70cdd5033d0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:05:50.558089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558038    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.558089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558085    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.558089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558095    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ab8c9-5117-43c6-be35-70cdd5033d0a-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.558089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558104    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.558089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558111    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.558089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558132    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlzjj\" (UniqueName: \"kubernetes.io/projected/918ab8c9-5117-43c6-be35-70cdd5033d0a-kube-api-access-hlzjj\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.558442 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558142    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.558442 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558150    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.558442 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.558158    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ab8c9-5117-43c6-be35-70cdd5033d0a-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:05:50.766838 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.766747    2563 generic.go:356] "Generic (PLEG): container finished" podID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerID="51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444" exitCode=0
Jul 17 19:05:50.766838 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.766793    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"
Jul 17 19:05:50.766838 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.766813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" event={"ID":"918ab8c9-5117-43c6-be35-70cdd5033d0a","Type":"ContainerDied","Data":"51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444"}
Jul 17 19:05:50.767061 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.766855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v" event={"ID":"918ab8c9-5117-43c6-be35-70cdd5033d0a","Type":"ContainerDied","Data":"067a515802b4c904c59daeebbe3d4cebd0ff38414aa6bddc802c9a658289bc36"}
Jul 17 19:05:50.767061 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.766876    2563 scope.go:117] "RemoveContainer" containerID="51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444"
Jul 17 19:05:50.777700 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.777682    2563 scope.go:117] "RemoveContainer" containerID="6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc"
Jul 17 19:05:50.785323 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.785305    2563 scope.go:117] "RemoveContainer" containerID="51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444"
Jul 17 19:05:50.785575 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:05:50.785556    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444\": container with ID starting with 51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444 not found: ID does not exist" containerID="51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444"
Jul 17 19:05:50.785643 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.785585    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444"} err="failed to get container status \"51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444\": rpc error: code = NotFound desc = could not find container \"51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444\": container with ID starting with 51d8029720788b27a78b89c9c1eff9c530e7fdef90074836dbd869aa0ebc4444 not found: ID does not exist"
Jul 17 19:05:50.785643 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.785610    2563 scope.go:117] "RemoveContainer" containerID="6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc"
Jul 17 19:05:50.785860 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:05:50.785841    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc\": container with ID starting with 6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc not found: ID does not exist" containerID="6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc"
Jul 17 19:05:50.785912 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.785866    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc"} err="failed to get container status \"6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc\": rpc error: code = NotFound desc = could not find container \"6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc\": container with ID starting with 6fbee3139e6c4436d9ebd95defec493ce0f5950b620d66d89c8cf73f093218fc not found: ID does not exist"
Jul 17 19:05:50.790924 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.790899    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"]
Jul 17 19:05:50.796674 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.796650    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7dc4fd53-openshift-default-986545fd7-qrp9v"]
Jul 17 19:05:50.922692 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:50.922664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:51.297326 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:51.297286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:51.693952 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:51.693922    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" path="/var/lib/kubelet/pods/918ab8c9-5117-43c6-be35-70cdd5033d0a/volumes"
Jul 17 19:05:57.034313 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.034277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:57.091542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.091504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:57.302977 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.302840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:57.420526 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.420489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:57.558628 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.558599    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"]
Jul 17 19:05:57.558903 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.558888    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy"
Jul 17 19:05:57.558956 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.558906    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy"
Jul 17 19:05:57.558990 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.558961    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy"
Jul 17 19:05:57.562112 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.562093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.565656 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.565635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0d4b3f70-gw-openshift-default-dockercfg-2gvt5\""
Jul 17 19:05:57.587493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.587469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"]
Jul 17 19:05:57.614666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-data\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.614809 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istiod-ca-cert\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.614809 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-socket\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.614809 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-credential-socket\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.614809 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-podinfo\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.614940 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-token\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.614940 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-certs\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.614940 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wns8p\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-kube-api-access-wns8p\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.614940 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.614924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-envoy\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.637223 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.637194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:57.716222 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istiod-ca-cert\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716222 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-socket\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-credential-socket\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-podinfo\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-token\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-certs\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wns8p\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-kube-api-access-wns8p\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-envoy\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-data\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716779 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-socket\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716779 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-credential-socket\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-certs\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.716940 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-data\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.717011 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.716991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istiod-ca-cert\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.718894 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.718872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-envoy\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.718980 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.718925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-podinfo\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.725880 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.725859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wns8p\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-kube-api-access-wns8p\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.726788 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.726769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-token\") pod \"e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") " pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:57.872617 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:57.872583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:58.018354 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.018326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"]
Jul 17 19:05:58.020376 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:05:58.020347    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e80ed6_a830_47fb_8c8a_009c38e6d2cb.slice/crio-5931857b34e2962a8de285513cf29e6841890397d01a253859959708ed6f5321 WatchSource:0}: Error finding container 5931857b34e2962a8de285513cf29e6841890397d01a253859959708ed6f5321: Status 404 returned error can't find the container with id 5931857b34e2962a8de285513cf29e6841890397d01a253859959708ed6f5321
Jul 17 19:05:58.022466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.022433    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:05:58.022520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.022505    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:05:58.022559 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.022535    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:05:58.800150 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.800093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" event={"ID":"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb","Type":"ContainerStarted","Data":"a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907"}
Jul 17 19:05:58.800576 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.800177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" event={"ID":"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb","Type":"ContainerStarted","Data":"5931857b34e2962a8de285513cf29e6841890397d01a253859959708ed6f5321"}
Jul 17 19:05:58.828012 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.827961    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" podStartSLOduration=1.827945115 podStartE2EDuration="1.827945115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:05:58.826399631 +0000 UTC m=+1967.737226395" watchObservedRunningTime="2026-07-17 19:05:58.827945115 +0000 UTC m=+1967.738771875"
Jul 17 19:05:58.873246 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.873209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:05:58.874376 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.874343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:58.945850 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.945820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"]
Jul 17 19:05:58.946139 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.946106    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy"
Jul 17 19:05:58.946139 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.946144    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy"
Jul 17 19:05:58.946327 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.946198    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="918ab8c9-5117-43c6-be35-70cdd5033d0a" containerName="istio-proxy"
Jul 17 19:05:58.948077 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.948060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:58.950539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.950517    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:05:58.950651 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.950613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0d4b3f70-serving-cert\""
Jul 17 19:05:58.950722 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.950661    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:05:58.950821 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.950804    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:05:58.960431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.960406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"]
Jul 17 19:05:58.988572 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:58.988540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:59.027969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.027936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6b2e2b-1dbc-4347-bba3-094d775c5680-openshift-service-ca\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.027969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.027973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsd5n\" (UniqueName: \"kubernetes.io/projected/ec6b2e2b-1dbc-4347-bba3-094d775c5680-kube-api-access-qsd5n\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.028195 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.028003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6b2e2b-1dbc-4347-bba3-094d775c5680-maas-api-tls\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.129349 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.129307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6b2e2b-1dbc-4347-bba3-094d775c5680-openshift-service-ca\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.129349 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.129357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsd5n\" (UniqueName: \"kubernetes.io/projected/ec6b2e2b-1dbc-4347-bba3-094d775c5680-kube-api-access-qsd5n\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.129562 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.129397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6b2e2b-1dbc-4347-bba3-094d775c5680-maas-api-tls\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.129980 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.129955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6b2e2b-1dbc-4347-bba3-094d775c5680-openshift-service-ca\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.131946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.131925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6b2e2b-1dbc-4347-bba3-094d775c5680-maas-api-tls\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.138886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.138859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsd5n\" (UniqueName: \"kubernetes.io/projected/ec6b2e2b-1dbc-4347-bba3-094d775c5680-kube-api-access-qsd5n\") pod \"maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.165618 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.165591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:59.259569 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.259534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:05:59.391493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.391414    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"]
Jul 17 19:05:59.394111 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:05:59.394084    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6b2e2b_1dbc_4347_bba3_094d775c5680.slice/crio-e636b2f364e393340ca5009cb72d83d216370e8d9f8d8e3ae11dbc4836ba87b7 WatchSource:0}: Error finding container e636b2f364e393340ca5009cb72d83d216370e8d9f8d8e3ae11dbc4836ba87b7: Status 404 returned error can't find the container with id e636b2f364e393340ca5009cb72d83d216370e8d9f8d8e3ae11dbc4836ba87b7
Jul 17 19:05:59.476778 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.476750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:59.577398 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.577371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:05:59.805158 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.805057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" event={"ID":"ec6b2e2b-1dbc-4347-bba3-094d775c5680","Type":"ContainerStarted","Data":"e636b2f364e393340ca5009cb72d83d216370e8d9f8d8e3ae11dbc4836ba87b7"}
Jul 17 19:05:59.880865 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:05:59.880836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:06:00.810360 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:00.810324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:06:00.811613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:00.811590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:06:01.811232 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:01.811143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:06:01.815224 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:01.815197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" event={"ID":"ec6b2e2b-1dbc-4347-bba3-094d775c5680","Type":"ContainerStarted","Data":"c7be6d76b356d901e1f5e6a1496ea986dbb862d39db48a289fdac50a259c8301"}
Jul 17 19:06:01.815365 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:01.815272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:06:01.832866 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:01.832812    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podStartSLOduration=1.698835724 podStartE2EDuration="3.832793963s" totalImagesPullingTime="2.133958239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:06:01.83263869 +0000 UTC m=+1970.743465470" watchObservedRunningTime="2026-07-17 19:06:01.832793963 +0000 UTC m=+1970.743620736"
Jul 17 19:06:02.821628 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:02.821597    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerID="c7be6d76b356d901e1f5e6a1496ea986dbb862d39db48a289fdac50a259c8301" exitCode=1
Jul 17 19:06:02.822057 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:02.821658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" event={"ID":"ec6b2e2b-1dbc-4347-bba3-094d775c5680","Type":"ContainerDied","Data":"c7be6d76b356d901e1f5e6a1496ea986dbb862d39db48a289fdac50a259c8301"}
Jul 17 19:06:02.822057 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:02.821893    2563 scope.go:117] "RemoveContainer" containerID="c7be6d76b356d901e1f5e6a1496ea986dbb862d39db48a289fdac50a259c8301"
Jul 17 19:06:03.827773 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:03.827685    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerID="ab6a9a26746973f6a19a103f07e324ade704e355f2df988c6913996501d39cf7" exitCode=1
Jul 17 19:06:03.827773 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:03.827723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" event={"ID":"ec6b2e2b-1dbc-4347-bba3-094d775c5680","Type":"ContainerDied","Data":"ab6a9a26746973f6a19a103f07e324ade704e355f2df988c6913996501d39cf7"}
Jul 17 19:06:03.827773 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:03.827771    2563 scope.go:117] "RemoveContainer" containerID="c7be6d76b356d901e1f5e6a1496ea986dbb862d39db48a289fdac50a259c8301"
Jul 17 19:06:03.828313 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:03.828028    2563 scope.go:117] "RemoveContainer" containerID="ab6a9a26746973f6a19a103f07e324ade704e355f2df988c6913996501d39cf7"
Jul 17 19:06:03.828313 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:03.828247    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:04.834039 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:04.833999    2563 scope.go:117] "RemoveContainer" containerID="ab6a9a26746973f6a19a103f07e324ade704e355f2df988c6913996501d39cf7"
Jul 17 19:06:04.834429 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:04.834189    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:06.445320 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:06.445288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:06:06.510689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:06.510647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:06:06.815953 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:06.815857    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:06:06.816305 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:06.816291    2563 scope.go:117] "RemoveContainer" containerID="ab6a9a26746973f6a19a103f07e324ade704e355f2df988c6913996501d39cf7"
Jul 17 19:06:06.816487 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:06.816461    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:09.260046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.260008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:06:09.260554 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.260393    2563 scope.go:117] "RemoveContainer" containerID="ab6a9a26746973f6a19a103f07e324ade704e355f2df988c6913996501d39cf7"
Jul 17 19:06:09.260603 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:09.260582    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:09.474743 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.474707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:06:09.856802 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.856768    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerID="a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907" exitCode=0
Jul 17 19:06:09.856980 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.856836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" event={"ID":"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb","Type":"ContainerDied","Data":"a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907"}
Jul 17 19:06:09.857284 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.857267    2563 scope.go:117] "RemoveContainer" containerID="a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907"
Jul 17 19:06:09.858226 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.858193    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:06:09.858282 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.858266    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:06:09.858319 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:09.858302    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:06:10.862915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:10.862883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" event={"ID":"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb","Type":"ContainerStarted","Data":"4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6"}
Jul 17 19:06:10.872978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:10.872951    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:06:10.873484 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:10.873199    2563 patch_prober.go:28] interesting pod/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused" start-of-body=
Jul 17 19:06:10.873484 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:10.873259    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused"
Jul 17 19:06:11.873438 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:11.873398    2563 patch_prober.go:28] interesting pod/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused" start-of-body=
Jul 17 19:06:11.873933 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:11.873464    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused"
Jul 17 19:06:12.873336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:12.873300    2563 patch_prober.go:28] interesting pod/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused" start-of-body=
Jul 17 19:06:12.873627 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:12.873598    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused"
Jul 17 19:06:13.873164 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:13.873095    2563 patch_prober.go:28] interesting pod/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused" start-of-body=
Jul 17 19:06:13.873356 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:13.873198    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused"
Jul 17 19:06:14.876724 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:14.876690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:06:14.877198 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:14.876985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:06:14.877864 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:14.877842    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:06:19.689353 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:19.689316    2563 scope.go:117] "RemoveContainer" containerID="ab6a9a26746973f6a19a103f07e324ade704e355f2df988c6913996501d39cf7"
Jul 17 19:06:20.903793 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:20.903760    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerID="2c0fe8fdbc7044e51ca61f17daf309ff0a90bba11010e682231abd28e0f4353f" exitCode=1
Jul 17 19:06:20.904190 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:20.903829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" event={"ID":"ec6b2e2b-1dbc-4347-bba3-094d775c5680","Type":"ContainerDied","Data":"2c0fe8fdbc7044e51ca61f17daf309ff0a90bba11010e682231abd28e0f4353f"}
Jul 17 19:06:20.904190 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:20.903890    2563 scope.go:117] "RemoveContainer" containerID="ab6a9a26746973f6a19a103f07e324ade704e355f2df988c6913996501d39cf7"
Jul 17 19:06:20.904320 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:20.904306    2563 scope.go:117] "RemoveContainer" containerID="2c0fe8fdbc7044e51ca61f17daf309ff0a90bba11010e682231abd28e0f4353f"
Jul 17 19:06:20.904561 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:20.904533    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:21.815563 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:21.815526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:06:21.915192 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:21.915165    2563 scope.go:117] "RemoveContainer" containerID="2c0fe8fdbc7044e51ca61f17daf309ff0a90bba11010e682231abd28e0f4353f"
Jul 17 19:06:21.915560 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:21.915342    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:29.260286 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:29.260247    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:06:29.260694 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:29.260680    2563 scope.go:117] "RemoveContainer" containerID="2c0fe8fdbc7044e51ca61f17daf309ff0a90bba11010e682231abd28e0f4353f"
Jul 17 19:06:29.260889 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:29.260855    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:44.689064 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:44.689024    2563 scope.go:117] "RemoveContainer" containerID="2c0fe8fdbc7044e51ca61f17daf309ff0a90bba11010e682231abd28e0f4353f"
Jul 17 19:06:46.005099 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:46.005064    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerID="54a02335cac4b09a3dcdae39fd7c14c606f11acfeec7fd5a981f062747d0065e" exitCode=1
Jul 17 19:06:46.005099 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:46.005103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" event={"ID":"ec6b2e2b-1dbc-4347-bba3-094d775c5680","Type":"ContainerDied","Data":"54a02335cac4b09a3dcdae39fd7c14c606f11acfeec7fd5a981f062747d0065e"}
Jul 17 19:06:46.005565 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:46.005174    2563 scope.go:117] "RemoveContainer" containerID="2c0fe8fdbc7044e51ca61f17daf309ff0a90bba11010e682231abd28e0f4353f"
Jul 17 19:06:46.005565 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:46.005503    2563 scope.go:117] "RemoveContainer" containerID="54a02335cac4b09a3dcdae39fd7c14c606f11acfeec7fd5a981f062747d0065e"
Jul 17 19:06:46.005729 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:46.005707    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:46.815560 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:46.815525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:06:47.010314 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:47.010283    2563 scope.go:117] "RemoveContainer" containerID="54a02335cac4b09a3dcdae39fd7c14c606f11acfeec7fd5a981f062747d0065e"
Jul 17 19:06:47.010671 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:47.010447    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:06:49.260566 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:49.260516    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:06:49.260956 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:06:49.260918    2563 scope.go:117] "RemoveContainer" containerID="54a02335cac4b09a3dcdae39fd7c14c606f11acfeec7fd5a981f062747d0065e"
Jul 17 19:06:49.261152 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:06:49.261104    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:07:03.438131 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:03.438091    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"]
Jul 17 19:07:03.438546 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:03.438340    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy" containerID="cri-o://4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6" gracePeriod=30
Jul 17 19:07:03.455611 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:03.455577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:03.584230 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:03.584202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:04.689858 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:04.689823    2563 scope.go:117] "RemoveContainer" containerID="54a02335cac4b09a3dcdae39fd7c14c606f11acfeec7fd5a981f062747d0065e"
Jul 17 19:07:04.690298 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:04.690016    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz_odh-ai-gateway-infra(ec6b2e2b-1dbc-4347-bba3-094d775c5680)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680"
Jul 17 19:07:06.507101 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:06.507063    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:07:08.568485 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.568444    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"]
Jul 17 19:07:08.600276 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.600245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:08.717847 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.717822    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:07:08.719964 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.719947    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:07:08.835920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.835878    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-credential-socket\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836087 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.835936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6b2e2b-1dbc-4347-bba3-094d775c5680-maas-api-tls\") pod \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") "
Jul 17 19:07:08.836087 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.835963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsd5n\" (UniqueName: \"kubernetes.io/projected/ec6b2e2b-1dbc-4347-bba3-094d775c5680-kube-api-access-qsd5n\") pod \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") "
Jul 17 19:07:08.836087 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.835995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-socket\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836087 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-podinfo\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836093    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-envoy\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836160    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istiod-ca-cert\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836188    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wns8p\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-kube-api-access-wns8p\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836213    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-token\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-certs\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836249    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:07:08.836352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836273    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6b2e2b-1dbc-4347-bba3-094d775c5680-openshift-service-ca\") pod \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\" (UID: \"ec6b2e2b-1dbc-4347-bba3-094d775c5680\") "
Jul 17 19:07:08.836352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-data\") pod \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\" (UID: \"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb\") "
Jul 17 19:07:08.836750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:07:08.836750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836651    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.836750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836674    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.836750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.836694    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:07:08.837138 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.837066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6b2e2b-1dbc-4347-bba3-094d775c5680-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec6b2e2b-1dbc-4347-bba3-094d775c5680" (UID: "ec6b2e2b-1dbc-4347-bba3-094d775c5680"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:07:08.837254 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.837231    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:07:08.837404 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.837376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:07:08.838893 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.838858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-kube-api-access-wns8p" (OuterVolumeSpecName: "kube-api-access-wns8p") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "kube-api-access-wns8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:07:08.839017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.838938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6b2e2b-1dbc-4347-bba3-094d775c5680-kube-api-access-qsd5n" (OuterVolumeSpecName: "kube-api-access-qsd5n") pod "ec6b2e2b-1dbc-4347-bba3-094d775c5680" (UID: "ec6b2e2b-1dbc-4347-bba3-094d775c5680"). InnerVolumeSpecName "kube-api-access-qsd5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:07:08.839017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.838996    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6b2e2b-1dbc-4347-bba3-094d775c5680-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec6b2e2b-1dbc-4347-bba3-094d775c5680" (UID: "ec6b2e2b-1dbc-4347-bba3-094d775c5680"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:07:08.839182 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.839136    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:07:08.839588 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.839565    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:07:08.839588 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.839573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" (UID: "a5e80ed6-a830-47fb-8c8a-009c38e6d2cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:07:08.937079 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937043    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937079 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937073    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wns8p\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-kube-api-access-wns8p\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937079 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937084    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937079 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937092    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937366 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937100    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6b2e2b-1dbc-4347-bba3-094d775c5680-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937366 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937108    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937366 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937144    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6b2e2b-1dbc-4347-bba3-094d775c5680-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937366 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937155    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsd5n\" (UniqueName: \"kubernetes.io/projected/ec6b2e2b-1dbc-4347-bba3-094d775c5680-kube-api-access-qsd5n\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937366 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937167    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:08.937366 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:08.937179    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:09.094727 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.094693    2563 generic.go:356] "Generic (PLEG): container finished" podID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerID="4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6" exitCode=0
Jul 17 19:07:09.094867 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.094736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" event={"ID":"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb","Type":"ContainerDied","Data":"4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6"}
Jul 17 19:07:09.094867 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.094781    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"
Jul 17 19:07:09.094867 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.094813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7" event={"ID":"a5e80ed6-a830-47fb-8c8a-009c38e6d2cb","Type":"ContainerDied","Data":"5931857b34e2962a8de285513cf29e6841890397d01a253859959708ed6f5321"}
Jul 17 19:07:09.094867 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.094841    2563 scope.go:117] "RemoveContainer" containerID="4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6"
Jul 17 19:07:09.096277 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.096257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz" event={"ID":"ec6b2e2b-1dbc-4347-bba3-094d775c5680","Type":"ContainerDied","Data":"e636b2f364e393340ca5009cb72d83d216370e8d9f8d8e3ae11dbc4836ba87b7"}
Jul 17 19:07:09.096362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.096285    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"
Jul 17 19:07:09.103532 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.103510    2563 scope.go:117] "RemoveContainer" containerID="a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907"
Jul 17 19:07:09.111591 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.111575    2563 scope.go:117] "RemoveContainer" containerID="4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6"
Jul 17 19:07:09.111835 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:09.111816    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6\": container with ID starting with 4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6 not found: ID does not exist" containerID="4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6"
Jul 17 19:07:09.111886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.111846    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6"} err="failed to get container status \"4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6\": rpc error: code = NotFound desc = could not find container \"4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6\": container with ID starting with 4305f02f5e9b514fc22a181827464472dca3bc92056bb7a1f61bd2070a6063e6 not found: ID does not exist"
Jul 17 19:07:09.111886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.111863    2563 scope.go:117] "RemoveContainer" containerID="a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907"
Jul 17 19:07:09.112065 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:09.112048    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907\": container with ID starting with a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907 not found: ID does not exist" containerID="a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907"
Jul 17 19:07:09.112104 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.112071    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907"} err="failed to get container status \"a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907\": rpc error: code = NotFound desc = could not find container \"a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907\": container with ID starting with a85bdce40b2a3f81849db8ce077ea59f2787fd645de30e8606889c8a4c8b0907 not found: ID does not exist"
Jul 17 19:07:09.112104 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.112085    2563 scope.go:117] "RemoveContainer" containerID="54a02335cac4b09a3dcdae39fd7c14c606f11acfeec7fd5a981f062747d0065e"
Jul 17 19:07:09.122770 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.122737    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"]
Jul 17 19:07:09.125595 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.125574    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0d4b3f70-557b7b469b-hhrbz"]
Jul 17 19:07:09.137053 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.137030    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"]
Jul 17 19:07:09.142024 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.142003    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0d4b3f70-gw-openshift-default-c7ffdd4d7-fnvm7"]
Jul 17 19:07:09.693505 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.693474    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" path="/var/lib/kubelet/pods/a5e80ed6-a830-47fb-8c8a-009c38e6d2cb/volumes"
Jul 17 19:07:09.693873 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:09.693847    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" path="/var/lib/kubelet/pods/ec6b2e2b-1dbc-4347-bba3-094d775c5680/volumes"
Jul 17 19:07:10.470155 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:10.470105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:15.153715 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153673    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"]
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153946    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153956    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153966    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153971    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153980    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153985    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153993    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.153998    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.154006    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.154011    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.154076    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.154083    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.154089    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.154094    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.154099    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6b2e2b-1dbc-4347-bba3-094d775c5680" containerName="maas-api"
Jul 17 19:07:15.154146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.154104    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy"
Jul 17 19:07:15.156903 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.156886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"
Jul 17 19:07:15.159402 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.159377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:07:15.159540 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.159398    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:07:15.165202 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.165175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"]
Jul 17 19:07:15.290721 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.290684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d6bb1a-15ac-4e26-9aae-9873313eb794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59\" (UID: \"69d6bb1a-15ac-4e26-9aae-9873313eb794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"
Jul 17 19:07:15.391720 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.391679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d6bb1a-15ac-4e26-9aae-9873313eb794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59\" (UID: \"69d6bb1a-15ac-4e26-9aae-9873313eb794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"
Jul 17 19:07:15.392327 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.392307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d6bb1a-15ac-4e26-9aae-9873313eb794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59\" (UID: \"69d6bb1a-15ac-4e26-9aae-9873313eb794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"
Jul 17 19:07:15.467763 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.467680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"
Jul 17 19:07:15.597701 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.597670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"]
Jul 17 19:07:15.600973 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:07:15.600947    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d6bb1a_15ac_4e26_9aae_9873313eb794.slice/crio-9c5c6cff0cc951a9f85ce8343f6936619b2c631347180308bee6155be5aaddaf WatchSource:0}: Error finding container 9c5c6cff0cc951a9f85ce8343f6936619b2c631347180308bee6155be5aaddaf: Status 404 returned error can't find the container with id 9c5c6cff0cc951a9f85ce8343f6936619b2c631347180308bee6155be5aaddaf
Jul 17 19:07:15.789689 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.789585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:15.853228 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.853188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:15.983520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.983487    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"]
Jul 17 19:07:15.983923 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.983908    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy"
Jul 17 19:07:15.983972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.983926    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e80ed6-a830-47fb-8c8a-009c38e6d2cb" containerName="istio-proxy"
Jul 17 19:07:15.988536 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.988514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:15.992607 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:15.992589    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-34da1ffa-openshift-default-dockercfg-2xzwf\""
Jul 17 19:07:16.001080 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.001041    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"]
Jul 17 19:07:16.027860 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.027834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:16.097716 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-envoy\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.097886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-certs\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.097886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22429772-b80c-4ec0-b708-a4c22ce79874-istiod-ca-cert\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.097886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-istio-token\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.097886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-data\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.097886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22429772-b80c-4ec0-b708-a4c22ce79874-istio-podinfo\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.098184 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxbj\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-kube-api-access-xkxbj\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.098184 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-socket\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.098184 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.097939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-credential-socket\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.125067 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.125030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" event={"ID":"69d6bb1a-15ac-4e26-9aae-9873313eb794","Type":"ContainerStarted","Data":"9c5c6cff0cc951a9f85ce8343f6936619b2c631347180308bee6155be5aaddaf"}
Jul 17 19:07:16.127114 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.127085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:16.199436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-envoy\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-certs\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22429772-b80c-4ec0-b708-a4c22ce79874-istiod-ca-cert\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-istio-token\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-data\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22429772-b80c-4ec0-b708-a4c22ce79874-istio-podinfo\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxbj\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-kube-api-access-xkxbj\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-socket\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-credential-socket\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.199972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.199945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-data\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.200364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.200007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-credential-socket\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.200364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.200040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-certs\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.200364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.200047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-socket\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.200364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.200320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22429772-b80c-4ec0-b708-a4c22ce79874-istiod-ca-cert\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.201913 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.201894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-envoy\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.202238 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.202210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22429772-b80c-4ec0-b708-a4c22ce79874-istio-podinfo\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.208024 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.207999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-istio-token\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.208528 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.208509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxbj\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-kube-api-access-xkxbj\") pod \"e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") " pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.299629 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.299595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:16.460231 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.460207    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"]
Jul 17 19:07:16.462379 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:07:16.462345    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22429772_b80c_4ec0_b708_a4c22ce79874.slice/crio-9eccd0769d2bcc011f161528647a117bbedb0f1c26b486a6ac279dd6d42b9cfd WatchSource:0}: Error finding container 9eccd0769d2bcc011f161528647a117bbedb0f1c26b486a6ac279dd6d42b9cfd: Status 404 returned error can't find the container with id 9eccd0769d2bcc011f161528647a117bbedb0f1c26b486a6ac279dd6d42b9cfd
Jul 17 19:07:16.464738 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.464693    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:07:16.464856 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.464774    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:07:16.464856 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:16.464830    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:07:17.132351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.132299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" event={"ID":"22429772-b80c-4ec0-b708-a4c22ce79874","Type":"ContainerStarted","Data":"50a722ac53f283a7be9d49c16815ba7b8935e4110f64294c0829e152dbd001d9"}
Jul 17 19:07:17.132537 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.132418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" event={"ID":"22429772-b80c-4ec0-b708-a4c22ce79874","Type":"ContainerStarted","Data":"9eccd0769d2bcc011f161528647a117bbedb0f1c26b486a6ac279dd6d42b9cfd"}
Jul 17 19:07:17.173338 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.172960    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" podStartSLOduration=2.172939222 podStartE2EDuration="2.172939222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:07:17.17017541 +0000 UTC m=+2046.081002162" watchObservedRunningTime="2026-07-17 19:07:17.172939222 +0000 UTC m=+2046.083765984"
Jul 17 19:07:17.268587 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.268533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:17.300716 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.300684    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:17.306245 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.306215    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:17.349514 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.349476    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"]
Jul 17 19:07:17.359702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.359674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.362575 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.362335    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:07:17.364817 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.362989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-34da1ffa-serving-cert\""
Jul 17 19:07:17.364817 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.363308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:07:17.364817 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.363677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"]
Jul 17 19:07:17.364817 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.364800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:17.462870 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.462768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:17.513227 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.513176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8764f532-06d3-4d99-b640-9eefdb36c7fe-openshift-service-ca\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.513399 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.513246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.513399 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.513265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prc8z\" (UniqueName: \"kubernetes.io/projected/8764f532-06d3-4d99-b640-9eefdb36c7fe-kube-api-access-prc8z\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.614036 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.613999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8764f532-06d3-4d99-b640-9eefdb36c7fe-openshift-service-ca\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.614229 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.614049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.614229 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.614072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prc8z\" (UniqueName: \"kubernetes.io/projected/8764f532-06d3-4d99-b640-9eefdb36c7fe-kube-api-access-prc8z\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.614229 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:17.614219    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-serving-cert: secret "maas-api-e2e-ait-34da1ffa-serving-cert" not found
Jul 17 19:07:17.614421 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:17.614284    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls podName:8764f532-06d3-4d99-b640-9eefdb36c7fe nodeName:}" failed. No retries permitted until 2026-07-17 19:07:18.114263472 +0000 UTC m=+2047.025090211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls") pod "maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" (UID: "8764f532-06d3-4d99-b640-9eefdb36c7fe") : secret "maas-api-e2e-ait-34da1ffa-serving-cert" not found
Jul 17 19:07:17.614773 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.614748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8764f532-06d3-4d99-b640-9eefdb36c7fe-openshift-service-ca\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.623848 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.623822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prc8z\" (UniqueName: \"kubernetes.io/projected/8764f532-06d3-4d99-b640-9eefdb36c7fe-kube-api-access-prc8z\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:17.701958 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:17.701926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:18.072707 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.072619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:18.119869 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.119832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:18.122412 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.122394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls\") pod \"maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:18.138153 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.138101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" event={"ID":"69d6bb1a-15ac-4e26-9aae-9873313eb794","Type":"ContainerStarted","Data":"5c52cdf9e60089c7c78ddd51b5b3de08cfd21bac10ef237503a94367ac2055a6"}
Jul 17 19:07:18.138292 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.138165    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:18.139180 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.139163    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:18.164045 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.163991    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" podStartSLOduration=1.067509216 podStartE2EDuration="3.163974439s" totalImagesPullingTime="2.096465223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:07:18.162716586 +0000 UTC m=+2047.073543384" watchObservedRunningTime="2026-07-17 19:07:18.163974439 +0000 UTC m=+2047.074801199"
Jul 17 19:07:18.167854 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.167825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:18.275239 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.275204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:18.413929 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:18.413849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"]
Jul 17 19:07:18.416054 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:07:18.416026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8764f532_06d3_4d99_b640_9eefdb36c7fe.slice/crio-14aa48d9a4dfddda6360675c34eedef8f3dc369ddbed4f8aa885b3ab8623ab27 WatchSource:0}: Error finding container 14aa48d9a4dfddda6360675c34eedef8f3dc369ddbed4f8aa885b3ab8623ab27: Status 404 returned error can't find the container with id 14aa48d9a4dfddda6360675c34eedef8f3dc369ddbed4f8aa885b3ab8623ab27
Jul 17 19:07:19.143466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:19.143437    2563 generic.go:356] "Generic (PLEG): container finished" podID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerID="5c52cdf9e60089c7c78ddd51b5b3de08cfd21bac10ef237503a94367ac2055a6" exitCode=6
Jul 17 19:07:19.143626 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:19.143524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" event={"ID":"69d6bb1a-15ac-4e26-9aae-9873313eb794","Type":"ContainerDied","Data":"5c52cdf9e60089c7c78ddd51b5b3de08cfd21bac10ef237503a94367ac2055a6"}
Jul 17 19:07:19.143717 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:19.143688    2563 scope.go:117] "RemoveContainer" containerID="5c52cdf9e60089c7c78ddd51b5b3de08cfd21bac10ef237503a94367ac2055a6"
Jul 17 19:07:19.145208 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:19.145179    2563 generic.go:356] "Generic (PLEG): container finished" podID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerID="c6c79b4331512a59cdb4629439fd73ac11a7b5d679f8ce5a48c9da467edfb0a9" exitCode=1
Jul 17 19:07:19.145321 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:19.145300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerDied","Data":"c6c79b4331512a59cdb4629439fd73ac11a7b5d679f8ce5a48c9da467edfb0a9"}
Jul 17 19:07:19.145408 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:19.145348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerStarted","Data":"14aa48d9a4dfddda6360675c34eedef8f3dc369ddbed4f8aa885b3ab8623ab27"}
Jul 17 19:07:19.145506 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:19.145490    2563 scope.go:117] "RemoveContainer" containerID="c6c79b4331512a59cdb4629439fd73ac11a7b5d679f8ce5a48c9da467edfb0a9"
Jul 17 19:07:19.400328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:19.400273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:07:20.152168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:20.152057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerStarted","Data":"39139af5c2dd2303b2b45aed06cd1925530b7d5021b72db688bccdb62fcaa19e"}
Jul 17 19:07:20.152541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:20.152240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:20.153971 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:20.153944    2563 generic.go:356] "Generic (PLEG): container finished" podID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerID="d2f434495d67217c4e9162dbab6756ce416bab811709fe478762635784fc80fb" exitCode=6
Jul 17 19:07:20.154112 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:20.154005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" event={"ID":"69d6bb1a-15ac-4e26-9aae-9873313eb794","Type":"ContainerDied","Data":"d2f434495d67217c4e9162dbab6756ce416bab811709fe478762635784fc80fb"}
Jul 17 19:07:20.154112 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:20.154053    2563 scope.go:117] "RemoveContainer" containerID="5c52cdf9e60089c7c78ddd51b5b3de08cfd21bac10ef237503a94367ac2055a6"
Jul 17 19:07:20.154249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:20.154174    2563 scope.go:117] "RemoveContainer" containerID="d2f434495d67217c4e9162dbab6756ce416bab811709fe478762635784fc80fb"
Jul 17 19:07:20.154371 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:20.154350    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59_odh-ai-gateway-infra(69d6bb1a-15ac-4e26-9aae-9873313eb794)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794"
Jul 17 19:07:20.187036 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:20.186990    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podStartSLOduration=1.9976396159999998 podStartE2EDuration="3.186973961s" totalImagesPullingTime="1.189334345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:07:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:07:20.184876122 +0000 UTC m=+2049.095702907" watchObservedRunningTime="2026-07-17 19:07:20.186973961 +0000 UTC m=+2049.097800719"
Jul 17 19:07:20.219959 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:20.219921    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:07:21.160334 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:21.160305    2563 scope.go:117] "RemoveContainer" containerID="d2f434495d67217c4e9162dbab6756ce416bab811709fe478762635784fc80fb"
Jul 17 19:07:21.160813 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:21.160565    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59_odh-ai-gateway-infra(69d6bb1a-15ac-4e26-9aae-9873313eb794)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794"
Jul 17 19:07:21.161846 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:21.161819    2563 generic.go:356] "Generic (PLEG): container finished" podID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerID="39139af5c2dd2303b2b45aed06cd1925530b7d5021b72db688bccdb62fcaa19e" exitCode=1
Jul 17 19:07:21.161962 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:21.161856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerDied","Data":"39139af5c2dd2303b2b45aed06cd1925530b7d5021b72db688bccdb62fcaa19e"}
Jul 17 19:07:21.161962 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:21.161901    2563 scope.go:117] "RemoveContainer" containerID="c6c79b4331512a59cdb4629439fd73ac11a7b5d679f8ce5a48c9da467edfb0a9"
Jul 17 19:07:21.162148 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:21.162107    2563 scope.go:117] "RemoveContainer" containerID="39139af5c2dd2303b2b45aed06cd1925530b7d5021b72db688bccdb62fcaa19e"
Jul 17 19:07:21.162336 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:21.162317    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:07:22.167316 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:22.167287    2563 scope.go:117] "RemoveContainer" containerID="39139af5c2dd2303b2b45aed06cd1925530b7d5021b72db688bccdb62fcaa19e"
Jul 17 19:07:22.167705 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:22.167449    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:07:28.276381 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:28.276345    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:28.276801 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:28.276725    2563 scope.go:117] "RemoveContainer" containerID="39139af5c2dd2303b2b45aed06cd1925530b7d5021b72db688bccdb62fcaa19e"
Jul 17 19:07:28.276950 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:28.276927    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:07:29.204472 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:29.204432    2563 generic.go:356] "Generic (PLEG): container finished" podID="22429772-b80c-4ec0-b708-a4c22ce79874" containerID="50a722ac53f283a7be9d49c16815ba7b8935e4110f64294c0829e152dbd001d9" exitCode=0
Jul 17 19:07:29.204665 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:29.204510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" event={"ID":"22429772-b80c-4ec0-b708-a4c22ce79874","Type":"ContainerDied","Data":"50a722ac53f283a7be9d49c16815ba7b8935e4110f64294c0829e152dbd001d9"}
Jul 17 19:07:29.204953 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:29.204935    2563 scope.go:117] "RemoveContainer" containerID="50a722ac53f283a7be9d49c16815ba7b8935e4110f64294c0829e152dbd001d9"
Jul 17 19:07:29.205971 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:29.205941    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:07:29.206065 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:29.206028    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:07:29.206153 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:29.206099    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:07:30.210361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:30.210327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" event={"ID":"22429772-b80c-4ec0-b708-a4c22ce79874","Type":"ContainerStarted","Data":"1a8ace1aeb8e3f91426c91a5ccdc7a9d41d0c63d41ef34d9a562fc2de859278c"}
Jul 17 19:07:30.300591 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:30.300551    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:30.301611 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:30.301581    2563 patch_prober.go:28] interesting pod/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Jul 17 19:07:30.301741 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:30.301643    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Jul 17 19:07:31.300388 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:31.300325    2563 patch_prober.go:28] interesting pod/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Jul 17 19:07:31.300825 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:31.300421    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Jul 17 19:07:32.300937 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:32.300897    2563 patch_prober.go:28] interesting pod/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Jul 17 19:07:32.301337 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:32.300967    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Jul 17 19:07:32.689680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:32.689644    2563 scope.go:117] "RemoveContainer" containerID="d2f434495d67217c4e9162dbab6756ce416bab811709fe478762635784fc80fb"
Jul 17 19:07:33.138623 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:33.138588    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:33.223752 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:33.223714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" event={"ID":"69d6bb1a-15ac-4e26-9aae-9873313eb794","Type":"ContainerStarted","Data":"42731f610d4efbff360741da6b15544e1c46956d8e3e60122f3814f0d9eb747f"}
Jul 17 19:07:33.301076 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:33.300976    2563 patch_prober.go:28] interesting pod/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Jul 17 19:07:33.301076 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:33.301038    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Jul 17 19:07:34.229190 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:34.229156    2563 generic.go:356] "Generic (PLEG): container finished" podID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerID="42731f610d4efbff360741da6b15544e1c46956d8e3e60122f3814f0d9eb747f" exitCode=6
Jul 17 19:07:34.229364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:34.229218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" event={"ID":"69d6bb1a-15ac-4e26-9aae-9873313eb794","Type":"ContainerDied","Data":"42731f610d4efbff360741da6b15544e1c46956d8e3e60122f3814f0d9eb747f"}
Jul 17 19:07:34.229364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:34.229287    2563 scope.go:117] "RemoveContainer" containerID="d2f434495d67217c4e9162dbab6756ce416bab811709fe478762635784fc80fb"
Jul 17 19:07:34.229569 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:34.229550    2563 scope.go:117] "RemoveContainer" containerID="42731f610d4efbff360741da6b15544e1c46956d8e3e60122f3814f0d9eb747f"
Jul 17 19:07:34.229771 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:34.229748    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59_odh-ai-gateway-infra(69d6bb1a-15ac-4e26-9aae-9873313eb794)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794"
Jul 17 19:07:34.257078 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:34.257046    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"]
Jul 17 19:07:34.300665 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:34.300626    2563 patch_prober.go:28] interesting pod/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Jul 17 19:07:34.300828 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:34.300692    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Jul 17 19:07:35.308960 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:35.308919    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:35.310258 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:35.310234    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:07:35.376850 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:35.376817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"
Jul 17 19:07:35.477956 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:35.477927    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d6bb1a-15ac-4e26-9aae-9873313eb794-maas-api-service-ca\") pod \"69d6bb1a-15ac-4e26-9aae-9873313eb794\" (UID: \"69d6bb1a-15ac-4e26-9aae-9873313eb794\") "
Jul 17 19:07:35.478288 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:35.478267    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d6bb1a-15ac-4e26-9aae-9873313eb794-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69d6bb1a-15ac-4e26-9aae-9873313eb794" (UID: "69d6bb1a-15ac-4e26-9aae-9873313eb794"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:07:35.578949 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:35.578878    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d6bb1a-15ac-4e26-9aae-9873313eb794-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:07:36.241001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:36.240966    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"
Jul 17 19:07:36.241188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:36.241022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59" event={"ID":"69d6bb1a-15ac-4e26-9aae-9873313eb794","Type":"ContainerDied","Data":"9c5c6cff0cc951a9f85ce8343f6936619b2c631347180308bee6155be5aaddaf"}
Jul 17 19:07:36.241188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:36.241081    2563 scope.go:117] "RemoveContainer" containerID="42731f610d4efbff360741da6b15544e1c46956d8e3e60122f3814f0d9eb747f"
Jul 17 19:07:36.256881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:36.256856    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"]
Jul 17 19:07:36.261683 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:36.261652    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-mhz59"]
Jul 17 19:07:37.694356 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:37.694324    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" path="/var/lib/kubelet/pods/69d6bb1a-15ac-4e26-9aae-9873313eb794/volumes"
Jul 17 19:07:39.689424 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:39.689393    2563 scope.go:117] "RemoveContainer" containerID="39139af5c2dd2303b2b45aed06cd1925530b7d5021b72db688bccdb62fcaa19e"
Jul 17 19:07:39.689798 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:39.689671    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:07:40.258796 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:40.258761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerStarted","Data":"51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486"}
Jul 17 19:07:40.258989 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:40.258974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:41.264875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:41.264840    2563 generic.go:356] "Generic (PLEG): container finished" podID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerID="51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486" exitCode=1
Jul 17 19:07:41.265355 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:41.264927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerDied","Data":"51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486"}
Jul 17 19:07:41.265355 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:41.264989    2563 scope.go:117] "RemoveContainer" containerID="39139af5c2dd2303b2b45aed06cd1925530b7d5021b72db688bccdb62fcaa19e"
Jul 17 19:07:41.265355 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:41.265260    2563 scope.go:117] "RemoveContainer" containerID="51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486"
Jul 17 19:07:41.265507 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:41.265447    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:07:42.273686 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:42.273658    2563 scope.go:117] "RemoveContainer" containerID="51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486"
Jul 17 19:07:42.274082 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:42.273838    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:07:48.275495 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:48.275458    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:07:48.275985 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:48.275888    2563 scope.go:117] "RemoveContainer" containerID="51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486"
Jul 17 19:07:48.276084 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:48.276061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:07:59.689631 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:07:59.689602    2563 scope.go:117] "RemoveContainer" containerID="51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486"
Jul 17 19:07:59.690019 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:07:59.689801    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:08:11.655549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:11.655520    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:08:11.657948 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:11.657928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:08:12.689500 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:12.689472    2563 scope.go:117] "RemoveContainer" containerID="51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486"
Jul 17 19:08:13.402768 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:13.402735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerStarted","Data":"6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"}
Jul 17 19:08:13.402950 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:13.402935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:08:14.409381 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:14.409343    2563 generic.go:356] "Generic (PLEG): container finished" podID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerID="6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d" exitCode=1
Jul 17 19:08:14.409867 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:14.409393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerDied","Data":"6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"}
Jul 17 19:08:14.409867 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:14.409444    2563 scope.go:117] "RemoveContainer" containerID="51e948a0807662f265ff31135ea4756c792cd4521d2e39e70d010b2f36828486"
Jul 17 19:08:14.409867 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:14.409704    2563 scope.go:117] "RemoveContainer" containerID="6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"
Jul 17 19:08:14.410031 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:08:14.409944    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:08:15.414533 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:15.414507    2563 scope.go:117] "RemoveContainer" containerID="6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"
Jul 17 19:08:15.415017 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:08:15.414731    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:08:18.276164 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:18.276101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:08:18.276591 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:18.276504    2563 scope.go:117] "RemoveContainer" containerID="6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"
Jul 17 19:08:18.276730 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:08:18.276710    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:08:29.689886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:29.689847    2563 scope.go:117] "RemoveContainer" containerID="6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"
Jul 17 19:08:29.690439 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:08:29.690090    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:08:41.691089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:41.691055    2563 scope.go:117] "RemoveContainer" containerID="6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"
Jul 17 19:08:41.691503 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:08:41.691304    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:08:55.694240 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:55.694212    2563 scope.go:117] "RemoveContainer" containerID="6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"
Jul 17 19:08:56.583791 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:56.583705    2563 generic.go:356] "Generic (PLEG): container finished" podID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6" exitCode=1
Jul 17 19:08:56.583791 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:56.583744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerDied","Data":"a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"}
Jul 17 19:08:56.584014 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:56.583796    2563 scope.go:117] "RemoveContainer" containerID="6154b22d5e949a301eee1f1ce6127aeb07b30e438b576fcfc2dc251b9e90482d"
Jul 17 19:08:56.584146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:56.584098    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:08:56.584391 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:08:56.584368    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:08:58.276500 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:58.276454    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:08:58.276919 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:58.276846    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:08:58.277039 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:08:58.277018    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:08:58.403484 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:58.403441    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:08:58.599791 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:08:58.599765    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:08:58.599964 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:08:58.599944    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:09:11.690832 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:09:11.690755    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:09:11.691323 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:09:11.690971    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:09:24.689361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:09:24.689325    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:09:24.689795 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:09:24.689580    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:09:36.688837 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:09:36.688804    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:09:36.689264 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:09:36.689011    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:09:49.689942 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:09:49.689907    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:09:49.690456 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:09:49.690152    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:10:00.689403 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:00.689372    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:10:00.689902 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:10:00.689550    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:10:11.690698 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:11.690664    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:10:11.691081 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:10:11.690899    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:10:22.689523 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:22.689489    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:10:23.920322 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:23.920235    2563 generic.go:356] "Generic (PLEG): container finished" podID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d" exitCode=1
Jul 17 19:10:23.920322 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:23.920302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerDied","Data":"8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"}
Jul 17 19:10:23.920757 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:23.920360    2563 scope.go:117] "RemoveContainer" containerID="a06a848f58cd745cfed6d62d4b22e8107bc0e4b4567506d2e95e80a59f30bff6"
Jul 17 19:10:23.920757 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:23.920644    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:10:23.920898 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:10:23.920877    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:10:28.275948 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:28.275902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:10:28.276449 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:28.276299    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:10:28.276519 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:10:28.276492    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:10:28.403218 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:28.403179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:10:28.940252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:28.940220    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:10:28.940431 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:10:28.940406    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:10:39.041912 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:39.041828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:10:39.126780 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:39.126730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:10:42.979312 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:42.979276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:10:43.081072 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:43.081037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:10:43.688915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:43.688884    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:10:43.689101 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:10:43.689076    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:10:56.689797 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:10:56.689768    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:10:56.690201 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:10:56.689951    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:11:00.101677 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:11:00.101640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:11:11.691028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:11:11.690997    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:11:11.691592 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:11:11.691221    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:11:23.689581 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:11:23.689546    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:11:23.689973 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:11:23.689729    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:11:34.688875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:11:34.688844    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:11:34.689266 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:11:34.689021    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:11:45.689932 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:11:45.689901    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:11:45.690418 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:11:45.690102    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:11:57.689603 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:11:57.689563    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:11:57.690096 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:11:57.689763    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:12:09.689923 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:12:09.689840    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:12:09.690345 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:12:09.690047    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:12:20.689475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:12:20.689440    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:12:20.689968 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:12:20.689648    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:12:35.689961 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:12:35.689916    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:12:35.692329 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:12:35.690189    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:12:44.306850 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:12:44.306798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:12:49.689774 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:12:49.689743    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:12:49.690198 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:12:49.689934    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:13:02.689841 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:02.689806    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:13:02.690337 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:02.689976    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:13:11.678290 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:11.678262    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:13:11.681858 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:11.681839    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:13:14.689477 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:14.689448    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:13:14.689853 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:14.689709    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:13:15.572400 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:15.572297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerStarted","Data":"dac9c8272b1a42fe21c163748df40aa066eb37a9e606d19e5805286b7834b023"}
Jul 17 19:13:15.572608 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:15.572587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:13:16.578316 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:16.578277    2563 generic.go:356] "Generic (PLEG): container finished" podID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerID="dac9c8272b1a42fe21c163748df40aa066eb37a9e606d19e5805286b7834b023" exitCode=1
Jul 17 19:13:16.578783 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:16.578361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerDied","Data":"dac9c8272b1a42fe21c163748df40aa066eb37a9e606d19e5805286b7834b023"}
Jul 17 19:13:16.578783 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:16.578451    2563 scope.go:117] "RemoveContainer" containerID="8404a5b07962bd4af23c63e820f8533b3753ec2cb6a254d09f5f51261cf6b08d"
Jul 17 19:13:16.578783 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:16.578670    2563 scope.go:117] "RemoveContainer" containerID="dac9c8272b1a42fe21c163748df40aa066eb37a9e606d19e5805286b7834b023"
Jul 17 19:13:16.578905 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:16.578881    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:13:17.584386 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:17.584355    2563 scope.go:117] "RemoveContainer" containerID="dac9c8272b1a42fe21c163748df40aa066eb37a9e606d19e5805286b7834b023"
Jul 17 19:13:17.584772 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:17.584576    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:13:18.276346 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:18.276307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:13:18.588337 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:18.588245    2563 scope.go:117] "RemoveContainer" containerID="dac9c8272b1a42fe21c163748df40aa066eb37a9e606d19e5805286b7834b023"
Jul 17 19:13:18.588771 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:18.588481    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7_odh-ai-gateway-infra(8764f532-06d3-4d99-b640-9eefdb36c7fe)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe"
Jul 17 19:13:22.483503 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:22.483468    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"]
Jul 17 19:13:22.483980 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:22.483680    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy" containerID="cri-o://1a8ace1aeb8e3f91426c91a5ccdc7a9d41d0c63d41ef34d9a562fc2de859278c" gracePeriod=30
Jul 17 19:13:22.502967 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:22.502935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:22.606509 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:22.606448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:25.218048 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:25.218004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:13:27.622893 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.622864    2563 generic.go:356] "Generic (PLEG): container finished" podID="22429772-b80c-4ec0-b708-a4c22ce79874" containerID="1a8ace1aeb8e3f91426c91a5ccdc7a9d41d0c63d41ef34d9a562fc2de859278c" exitCode=0
Jul 17 19:13:27.623312 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.622935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" event={"ID":"22429772-b80c-4ec0-b708-a4c22ce79874","Type":"ContainerDied","Data":"1a8ace1aeb8e3f91426c91a5ccdc7a9d41d0c63d41ef34d9a562fc2de859278c"}
Jul 17 19:13:27.623312 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.622996    2563 scope.go:117] "RemoveContainer" containerID="50a722ac53f283a7be9d49c16815ba7b8935e4110f64294c0829e152dbd001d9"
Jul 17 19:13:27.733320 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.733296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:13:27.799707 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799677    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-istio-token\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.799889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-envoy\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.799889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-credential-socket\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.799889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799810    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkxbj\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-kube-api-access-xkxbj\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.799889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-data\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.799889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799876    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-certs\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.800179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799898    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22429772-b80c-4ec0-b708-a4c22ce79874-istiod-ca-cert\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.800179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22429772-b80c-4ec0-b708-a4c22ce79874-istio-podinfo\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.800179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.799958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-socket\") pod \"22429772-b80c-4ec0-b708-a4c22ce79874\" (UID: \"22429772-b80c-4ec0-b708-a4c22ce79874\") "
Jul 17 19:13:27.800179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.800102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:13:27.800391 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.800311    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:27.800391 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.800345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:13:27.800391 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.800355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22429772-b80c-4ec0-b708-a4c22ce79874-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:13:27.800602 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.800496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:13:27.800874 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.800849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-data" (OuterVolumeSpecName: "istio-data") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:13:27.802911 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.802883    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:13:27.802911 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.802890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-istio-token" (OuterVolumeSpecName: "istio-token") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:13:27.803039 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.802982    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22429772-b80c-4ec0-b708-a4c22ce79874-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:13:27.803140 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.803100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-kube-api-access-xkxbj" (OuterVolumeSpecName: "kube-api-access-xkxbj") pod "22429772-b80c-4ec0-b708-a4c22ce79874" (UID: "22429772-b80c-4ec0-b708-a4c22ce79874"). InnerVolumeSpecName "kube-api-access-xkxbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:13:27.900990 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.900951    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkxbj\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-kube-api-access-xkxbj\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:27.900990 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.900984    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:27.900990 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.900993    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:27.900990 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.901001    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22429772-b80c-4ec0-b708-a4c22ce79874-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:27.901286 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.901012    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22429772-b80c-4ec0-b708-a4c22ce79874-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:27.901286 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.901020    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:27.901286 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.901029    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22429772-b80c-4ec0-b708-a4c22ce79874-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:27.901286 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:27.901038    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22429772-b80c-4ec0-b708-a4c22ce79874-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:28.146262 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.146224    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"]
Jul 17 19:13:28.168862 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.168829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:28.269461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.269435    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:13:28.405096 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.404997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8764f532-06d3-4d99-b640-9eefdb36c7fe-openshift-service-ca\") pod \"8764f532-06d3-4d99-b640-9eefdb36c7fe\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") "
Jul 17 19:13:28.405096 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.405048    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls\") pod \"8764f532-06d3-4d99-b640-9eefdb36c7fe\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") "
Jul 17 19:13:28.405096 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.405066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prc8z\" (UniqueName: \"kubernetes.io/projected/8764f532-06d3-4d99-b640-9eefdb36c7fe-kube-api-access-prc8z\") pod \"8764f532-06d3-4d99-b640-9eefdb36c7fe\" (UID: \"8764f532-06d3-4d99-b640-9eefdb36c7fe\") "
Jul 17 19:13:28.405488 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.405460    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8764f532-06d3-4d99-b640-9eefdb36c7fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8764f532-06d3-4d99-b640-9eefdb36c7fe" (UID: "8764f532-06d3-4d99-b640-9eefdb36c7fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:13:28.407369 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.407345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8764f532-06d3-4d99-b640-9eefdb36c7fe-kube-api-access-prc8z" (OuterVolumeSpecName: "kube-api-access-prc8z") pod "8764f532-06d3-4d99-b640-9eefdb36c7fe" (UID: "8764f532-06d3-4d99-b640-9eefdb36c7fe"). InnerVolumeSpecName "kube-api-access-prc8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:13:28.407458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.407370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8764f532-06d3-4d99-b640-9eefdb36c7fe" (UID: "8764f532-06d3-4d99-b640-9eefdb36c7fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:13:28.506261 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.506228    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8764f532-06d3-4d99-b640-9eefdb36c7fe-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:28.506261 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.506262    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prc8z\" (UniqueName: \"kubernetes.io/projected/8764f532-06d3-4d99-b640-9eefdb36c7fe-kube-api-access-prc8z\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:28.506444 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.506273    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8764f532-06d3-4d99-b640-9eefdb36c7fe-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:28.627965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.627935    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"
Jul 17 19:13:28.628545 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.627930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7" event={"ID":"8764f532-06d3-4d99-b640-9eefdb36c7fe","Type":"ContainerDied","Data":"14aa48d9a4dfddda6360675c34eedef8f3dc369ddbed4f8aa885b3ab8623ab27"}
Jul 17 19:13:28.628545 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.628069    2563 scope.go:117] "RemoveContainer" containerID="dac9c8272b1a42fe21c163748df40aa066eb37a9e606d19e5805286b7834b023"
Jul 17 19:13:28.629369 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.629293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf" event={"ID":"22429772-b80c-4ec0-b708-a4c22ce79874","Type":"ContainerDied","Data":"9eccd0769d2bcc011f161528647a117bbedb0f1c26b486a6ac279dd6d42b9cfd"}
Jul 17 19:13:28.629369 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.629322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"
Jul 17 19:13:28.637729 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.637709    2563 scope.go:117] "RemoveContainer" containerID="1a8ace1aeb8e3f91426c91a5ccdc7a9d41d0c63d41ef34d9a562fc2de859278c"
Jul 17 19:13:28.654672 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.654646    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"]
Jul 17 19:13:28.659431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.659366    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34da1ffa-98b9f7f5f-w2rd7"]
Jul 17 19:13:28.672906 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.672880    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"]
Jul 17 19:13:28.677080 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:28.677058    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-34da1ffa-openshift-default-568bb55c79-498wf"]
Jul 17 19:13:29.269961 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:29.269925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:29.693959 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:29.693925    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" path="/var/lib/kubelet/pods/22429772-b80c-4ec0-b708-a4c22ce79874/volumes"
Jul 17 19:13:29.694363 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:29.694348    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" path="/var/lib/kubelet/pods/8764f532-06d3-4d99-b640-9eefdb36c7fe/volumes"
Jul 17 19:13:33.638143 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638037    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"]
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638354    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638366    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638375    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638380    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638390    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638395    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638401    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638406    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638422    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638428    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638436    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638440    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638446    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638451    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638458    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638463    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638508    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638514    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638520    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638526    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638532    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638538    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638543    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638549    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.638542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638554    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:33.639413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638558    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:33.639413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.638564    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22429772-b80c-4ec0-b708-a4c22ce79874" containerName="istio-proxy"
Jul 17 19:13:33.643044 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.643025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"
Jul 17 19:13:33.645841 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.645809    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:13:33.645841 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.645824    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:13:33.650688 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.650336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"]
Jul 17 19:13:33.749478 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.749444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8573d2b7-6cdb-4051-8846-d77786fa1b46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9\" (UID: \"8573d2b7-6cdb-4051-8846-d77786fa1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"
Jul 17 19:13:33.850461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.850419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8573d2b7-6cdb-4051-8846-d77786fa1b46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9\" (UID: \"8573d2b7-6cdb-4051-8846-d77786fa1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"
Jul 17 19:13:33.851070 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.851047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8573d2b7-6cdb-4051-8846-d77786fa1b46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9\" (UID: \"8573d2b7-6cdb-4051-8846-d77786fa1b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"
Jul 17 19:13:33.955526 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:33.955411    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"
Jul 17 19:13:34.091700 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:34.091673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"]
Jul 17 19:13:34.094370 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:13:34.094341    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8573d2b7_6cdb_4051_8846_d77786fa1b46.slice/crio-01763fe6888f9f4f13d4fea3efcfa8996daab7c651df5f634d61738884a4d4c8 WatchSource:0}: Error finding container 01763fe6888f9f4f13d4fea3efcfa8996daab7c651df5f634d61738884a4d4c8: Status 404 returned error can't find the container with id 01763fe6888f9f4f13d4fea3efcfa8996daab7c651df5f634d61738884a4d4c8
Jul 17 19:13:34.654562 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:34.654524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" event={"ID":"8573d2b7-6cdb-4051-8846-d77786fa1b46","Type":"ContainerStarted","Data":"91b07e648d523c9f4ef860372d9bdc61ef1f4058c3a7ec14d42d32288fd7217d"}
Jul 17 19:13:34.655028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:34.654580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" event={"ID":"8573d2b7-6cdb-4051-8846-d77786fa1b46","Type":"ContainerStarted","Data":"01763fe6888f9f4f13d4fea3efcfa8996daab7c651df5f634d61738884a4d4c8"}
Jul 17 19:13:34.669646 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:34.669584    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" podStartSLOduration=1.6695658519999999 podStartE2EDuration="1.669565852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:13:34.669098009 +0000 UTC m=+2423.579924774" watchObservedRunningTime="2026-07-17 19:13:34.669565852 +0000 UTC m=+2423.580392615"
Jul 17 19:13:35.015816 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.015735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:35.115812 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.115784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:35.244725 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.244694    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"]
Jul 17 19:13:35.244984 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.244973    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:35.245035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.244986    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:35.245035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.244999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:35.245035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.245005    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:35.245035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.245016    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:35.245035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.245022    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:35.245035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.245033    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:35.245035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.245038    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d6bb1a-15ac-4e26-9aae-9873313eb794" containerName="revoke-keys"
Jul 17 19:13:35.245270 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.245080    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8764f532-06d3-4d99-b640-9eefdb36c7fe" containerName="maas-api"
Jul 17 19:13:35.248023 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.248000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.250785 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.250766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b74adac0-openshift-default-dockercfg-lxswq\""
Jul 17 19:13:35.259351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.259328    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"]
Jul 17 19:13:35.274622 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.274553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:35.365306 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-socket\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.365306 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-certs\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.365539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5jz\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-kube-api-access-gt5jz\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.365539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-podinfo\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.365539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88718a3f-1d10-46c8-91a9-b4431926eb63-istiod-ca-cert\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.365539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-credential-socket\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.365539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-envoy\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.365539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-token\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.365767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.365579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-data\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.466690 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.466654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-credential-socket\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.466690 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.466687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-envoy\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.466946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.466816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-token\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.466946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.466869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-data\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.466946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.466927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-socket\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467187 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.466951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-certs\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467187 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.467053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-credential-socket\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467303 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.467198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-data\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467303 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.467236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5jz\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-kube-api-access-gt5jz\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467303 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.467297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-podinfo\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467451 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.467392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88718a3f-1d10-46c8-91a9-b4431926eb63-istiod-ca-cert\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467504 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.467473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-socket\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467644 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.467618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-certs\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.467966 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.467945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88718a3f-1d10-46c8-91a9-b4431926eb63-istiod-ca-cert\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.469075 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.469057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-envoy\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.469626 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.469608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-podinfo\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.475365 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.475325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-token\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.475830 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.475808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5jz\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-kube-api-access-gt5jz\") pod \"e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") " pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.560144 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.560036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:35.661736 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.661697    2563 generic.go:356] "Generic (PLEG): container finished" podID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerID="91b07e648d523c9f4ef860372d9bdc61ef1f4058c3a7ec14d42d32288fd7217d" exitCode=6
Jul 17 19:13:35.661736 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.661740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" event={"ID":"8573d2b7-6cdb-4051-8846-d77786fa1b46","Type":"ContainerDied","Data":"91b07e648d523c9f4ef860372d9bdc61ef1f4058c3a7ec14d42d32288fd7217d"}
Jul 17 19:13:35.662382 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.662018    2563 scope.go:117] "RemoveContainer" containerID="91b07e648d523c9f4ef860372d9bdc61ef1f4058c3a7ec14d42d32288fd7217d"
Jul 17 19:13:35.697969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.697947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"]
Jul 17 19:13:35.700541 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:13:35.700513    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88718a3f_1d10_46c8_91a9_b4431926eb63.slice/crio-bd4d2afe215af222970f541ece9051ea2db5a24c4014abf56366b699faea33b7 WatchSource:0}: Error finding container bd4d2afe215af222970f541ece9051ea2db5a24c4014abf56366b699faea33b7: Status 404 returned error can't find the container with id bd4d2afe215af222970f541ece9051ea2db5a24c4014abf56366b699faea33b7
Jul 17 19:13:35.702607 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.702576    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:13:35.702669 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.702646    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:13:35.702707 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:35.702674    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:13:36.417736 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.417694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:36.513432 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.513403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:36.606042 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.606001    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"]
Jul 17 19:13:36.609363 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.609342    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:36.612172 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.612143    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b74adac0-serving-cert\""
Jul 17 19:13:36.612298 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.612113    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:13:36.613030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.613014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:13:36.619527 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.619501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"]
Jul 17 19:13:36.639256 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.639227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:36.668354 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.668322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" event={"ID":"88718a3f-1d10-46c8-91a9-b4431926eb63","Type":"ContainerStarted","Data":"9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc"}
Jul 17 19:13:36.668800 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.668374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" event={"ID":"88718a3f-1d10-46c8-91a9-b4431926eb63","Type":"ContainerStarted","Data":"bd4d2afe215af222970f541ece9051ea2db5a24c4014abf56366b699faea33b7"}
Jul 17 19:13:36.670184 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.670151    2563 generic.go:356] "Generic (PLEG): container finished" podID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerID="d5920f164465a298a6fd02a7076f2daf53a6a9fd8acf48c82f088f8fe77d43c7" exitCode=6
Jul 17 19:13:36.670303 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.670225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" event={"ID":"8573d2b7-6cdb-4051-8846-d77786fa1b46","Type":"ContainerDied","Data":"d5920f164465a298a6fd02a7076f2daf53a6a9fd8acf48c82f088f8fe77d43c7"}
Jul 17 19:13:36.670303 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.670272    2563 scope.go:117] "RemoveContainer" containerID="91b07e648d523c9f4ef860372d9bdc61ef1f4058c3a7ec14d42d32288fd7217d"
Jul 17 19:13:36.670448 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.670431    2563 scope.go:117] "RemoveContainer" containerID="d5920f164465a298a6fd02a7076f2daf53a6a9fd8acf48c82f088f8fe77d43c7"
Jul 17 19:13:36.670649 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:36.670630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9_odh-ai-gateway-infra(8573d2b7-6cdb-4051-8846-d77786fa1b46)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46"
Jul 17 19:13:36.690467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.690420    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" podStartSLOduration=1.690405173 podStartE2EDuration="1.690405173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:13:36.688543888 +0000 UTC m=+2425.599370676" watchObservedRunningTime="2026-07-17 19:13:36.690405173 +0000 UTC m=+2425.601231925"
Jul 17 19:13:36.778297 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.778256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:36.778513 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.778435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vbvh\" (UniqueName: \"kubernetes.io/projected/8a1a84cd-b9ba-4005-9642-ae46786aee6d-kube-api-access-7vbvh\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:36.778580 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.778517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1a84cd-b9ba-4005-9642-ae46786aee6d-openshift-service-ca\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:36.879771 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.879692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:36.879920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.879780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vbvh\" (UniqueName: \"kubernetes.io/projected/8a1a84cd-b9ba-4005-9642-ae46786aee6d-kube-api-access-7vbvh\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:36.879920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.879812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1a84cd-b9ba-4005-9642-ae46786aee6d-openshift-service-ca\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:36.879920 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:36.879843    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-serving-cert: secret "maas-api-e2e-derive-b74adac0-serving-cert" not found
Jul 17 19:13:36.879920 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:36.879907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls podName:8a1a84cd-b9ba-4005-9642-ae46786aee6d nodeName:}" failed. No retries permitted until 2026-07-17 19:13:37.379891341 +0000 UTC m=+2426.290718080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls") pod "maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" (UID: "8a1a84cd-b9ba-4005-9642-ae46786aee6d") : secret "maas-api-e2e-derive-b74adac0-serving-cert" not found
Jul 17 19:13:36.880486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.880460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1a84cd-b9ba-4005-9642-ae46786aee6d-openshift-service-ca\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:36.888761 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:36.888739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vbvh\" (UniqueName: \"kubernetes.io/projected/8a1a84cd-b9ba-4005-9642-ae46786aee6d-kube-api-access-7vbvh\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:37.128733 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.128696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:37.233509 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.233425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:37.329802 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.329767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:37.384766 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.384716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:37.387341 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.387321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls\") pod \"maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:37.521742 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.521642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:37.560979 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.560947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:37.566454 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.566427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:37.658939 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.658913    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"]
Jul 17 19:13:37.661448 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:13:37.661416    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1a84cd_b9ba_4005_9642_ae46786aee6d.slice/crio-51fb8610474114be376adb0ffdaef02194c74a3bdd02cd274cdb6a58a348c5ff WatchSource:0}: Error finding container 51fb8610474114be376adb0ffdaef02194c74a3bdd02cd274cdb6a58a348c5ff: Status 404 returned error can't find the container with id 51fb8610474114be376adb0ffdaef02194c74a3bdd02cd274cdb6a58a348c5ff
Jul 17 19:13:37.675876 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.675843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" event={"ID":"8a1a84cd-b9ba-4005-9642-ae46786aee6d","Type":"ContainerStarted","Data":"51fb8610474114be376adb0ffdaef02194c74a3bdd02cd274cdb6a58a348c5ff"}
Jul 17 19:13:37.677598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.677569    2563 scope.go:117] "RemoveContainer" containerID="d5920f164465a298a6fd02a7076f2daf53a6a9fd8acf48c82f088f8fe77d43c7"
Jul 17 19:13:37.677701 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.677598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:37.677805 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:37.677783    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9_odh-ai-gateway-infra(8573d2b7-6cdb-4051-8846-d77786fa1b46)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46"
Jul 17 19:13:37.678496 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:37.678478    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:38.682487 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:38.682447    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerID="e4b9a7464c806189b158f9eb5af43772e64daa9454fb481e5c7df2111855cff4" exitCode=1
Jul 17 19:13:38.682866 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:38.682534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" event={"ID":"8a1a84cd-b9ba-4005-9642-ae46786aee6d","Type":"ContainerDied","Data":"e4b9a7464c806189b158f9eb5af43772e64daa9454fb481e5c7df2111855cff4"}
Jul 17 19:13:38.682866 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:38.682789    2563 scope.go:117] "RemoveContainer" containerID="e4b9a7464c806189b158f9eb5af43772e64daa9454fb481e5c7df2111855cff4"
Jul 17 19:13:39.689264 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:39.689224    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerID="b243093c2c5b6093de0fbd7332c53187093b13049c5b17942659c453b3a831dd" exitCode=1
Jul 17 19:13:39.689725 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:39.689643    2563 scope.go:117] "RemoveContainer" containerID="b243093c2c5b6093de0fbd7332c53187093b13049c5b17942659c453b3a831dd"
Jul 17 19:13:39.689863 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:39.689844    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:13:39.693963 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:39.693936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" event={"ID":"8a1a84cd-b9ba-4005-9642-ae46786aee6d","Type":"ContainerDied","Data":"b243093c2c5b6093de0fbd7332c53187093b13049c5b17942659c453b3a831dd"}
Jul 17 19:13:39.694104 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:39.694006    2563 scope.go:117] "RemoveContainer" containerID="e4b9a7464c806189b158f9eb5af43772e64daa9454fb481e5c7df2111855cff4"
Jul 17 19:13:39.814967 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:39.814873    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:13:40.317557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:40.317524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:40.388219 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:40.388185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:40.698257 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:40.698222    2563 scope.go:117] "RemoveContainer" containerID="b243093c2c5b6093de0fbd7332c53187093b13049c5b17942659c453b3a831dd"
Jul 17 19:13:40.698765 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:40.698465    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:13:42.522098 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:42.522054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:42.522539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:42.522520    2563 scope.go:117] "RemoveContainer" containerID="b243093c2c5b6093de0fbd7332c53187093b13049c5b17942659c453b3a831dd"
Jul 17 19:13:42.522705 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:42.522684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:13:44.295819 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:44.295755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:44.397634 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:44.397605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:47.522813 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:47.522699    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:13:47.523264 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:47.523196    2563 scope.go:117] "RemoveContainer" containerID="b243093c2c5b6093de0fbd7332c53187093b13049c5b17942659c453b3a831dd"
Jul 17 19:13:47.523462 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:47.523437    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:13:47.728569 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:47.728535    2563 generic.go:356] "Generic (PLEG): container finished" podID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerID="9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc" exitCode=0
Jul 17 19:13:47.728750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:47.728604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" event={"ID":"88718a3f-1d10-46c8-91a9-b4431926eb63","Type":"ContainerDied","Data":"9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc"}
Jul 17 19:13:47.728993 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:47.728979    2563 scope.go:117] "RemoveContainer" containerID="9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc"
Jul 17 19:13:47.730001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:47.729972    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:13:47.730104 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:47.730054    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:13:47.730203 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:47.730101    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:13:48.291394 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:48.291354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:13:48.734976 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:48.734939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" event={"ID":"88718a3f-1d10-46c8-91a9-b4431926eb63","Type":"ContainerStarted","Data":"7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec"}
Jul 17 19:13:49.560703 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:49.560655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:49.561980 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:49.561953    2563 patch_prober.go:28] interesting pod/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 17 19:13:49.562133 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:49.562006    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 17 19:13:50.561168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:50.561104    2563 patch_prober.go:28] interesting pod/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 17 19:13:50.561589 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:50.561213    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 17 19:13:51.560841 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:51.560803    2563 patch_prober.go:28] interesting pod/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 17 19:13:51.561041 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:51.560860    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 17 19:13:51.691943 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:51.691913    2563 scope.go:117] "RemoveContainer" containerID="d5920f164465a298a6fd02a7076f2daf53a6a9fd8acf48c82f088f8fe77d43c7"
Jul 17 19:13:52.561056 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:52.561017    2563 patch_prober.go:28] interesting pod/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 17 19:13:52.561263 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:52.561073    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 17 19:13:52.678362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:52.678330    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:52.752334 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:52.752304    2563 generic.go:356] "Generic (PLEG): container finished" podID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerID="4e74a9e72e5f4bf0862b9863a0b828772f71ced33e5b05451af2423b1b8d3726" exitCode=6
Jul 17 19:13:52.752745 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:52.752368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" event={"ID":"8573d2b7-6cdb-4051-8846-d77786fa1b46","Type":"ContainerDied","Data":"4e74a9e72e5f4bf0862b9863a0b828772f71ced33e5b05451af2423b1b8d3726"}
Jul 17 19:13:52.752745 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:52.752424    2563 scope.go:117] "RemoveContainer" containerID="d5920f164465a298a6fd02a7076f2daf53a6a9fd8acf48c82f088f8fe77d43c7"
Jul 17 19:13:52.752745 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:52.752704    2563 scope.go:117] "RemoveContainer" containerID="4e74a9e72e5f4bf0862b9863a0b828772f71ced33e5b05451af2423b1b8d3726"
Jul 17 19:13:52.752937 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:13:52.752914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9_odh-ai-gateway-infra(8573d2b7-6cdb-4051-8846-d77786fa1b46)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46"
Jul 17 19:13:53.565353 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:53.565322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:53.566231 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:53.566210    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:13:53.782920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:53.779352    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"]
Jul 17 19:13:53.914562 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:53.914536    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"
Jul 17 19:13:53.938614 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:53.938583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8573d2b7-6cdb-4051-8846-d77786fa1b46-maas-api-service-ca\") pod \"8573d2b7-6cdb-4051-8846-d77786fa1b46\" (UID: \"8573d2b7-6cdb-4051-8846-d77786fa1b46\") "
Jul 17 19:13:53.938916 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:53.938893    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8573d2b7-6cdb-4051-8846-d77786fa1b46-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8573d2b7-6cdb-4051-8846-d77786fa1b46" (UID: "8573d2b7-6cdb-4051-8846-d77786fa1b46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:13:54.039910 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:54.039879    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8573d2b7-6cdb-4051-8846-d77786fa1b46-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:13:54.762351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:54.762317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"
Jul 17 19:13:54.762351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:54.762334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9" event={"ID":"8573d2b7-6cdb-4051-8846-d77786fa1b46","Type":"ContainerDied","Data":"01763fe6888f9f4f13d4fea3efcfa8996daab7c651df5f634d61738884a4d4c8"}
Jul 17 19:13:54.762586 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:54.762386    2563 scope.go:117] "RemoveContainer" containerID="4e74a9e72e5f4bf0862b9863a0b828772f71ced33e5b05451af2423b1b8d3726"
Jul 17 19:13:54.786842 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:54.786810    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"]
Jul 17 19:13:54.790282 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:54.790257    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7f5k9"]
Jul 17 19:13:55.693334 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:13:55.693299    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" path="/var/lib/kubelet/pods/8573d2b7-6cdb-4051-8846-d77786fa1b46/volumes"
Jul 17 19:14:01.692474 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:01.692444    2563 scope.go:117] "RemoveContainer" containerID="b243093c2c5b6093de0fbd7332c53187093b13049c5b17942659c453b3a831dd"
Jul 17 19:14:02.800491 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:02.800399    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerID="be6bb399c118f4cc79d3ebdd603a593c6fc3c1075b6de1691b414ba041c7fc31" exitCode=1
Jul 17 19:14:02.800491 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:02.800476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" event={"ID":"8a1a84cd-b9ba-4005-9642-ae46786aee6d","Type":"ContainerDied","Data":"be6bb399c118f4cc79d3ebdd603a593c6fc3c1075b6de1691b414ba041c7fc31"}
Jul 17 19:14:02.800941 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:02.800544    2563 scope.go:117] "RemoveContainer" containerID="b243093c2c5b6093de0fbd7332c53187093b13049c5b17942659c453b3a831dd"
Jul 17 19:14:02.800941 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:02.800869    2563 scope.go:117] "RemoveContainer" containerID="be6bb399c118f4cc79d3ebdd603a593c6fc3c1075b6de1691b414ba041c7fc31"
Jul 17 19:14:02.801091 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:14:02.801066    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:14:07.522456 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:07.522421    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:14:07.522963 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:07.522513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:14:07.522963 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:07.522835    2563 scope.go:117] "RemoveContainer" containerID="be6bb399c118f4cc79d3ebdd603a593c6fc3c1075b6de1691b414ba041c7fc31"
Jul 17 19:14:07.523038 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:14:07.522993    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:14:20.689206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:20.689170    2563 scope.go:117] "RemoveContainer" containerID="be6bb399c118f4cc79d3ebdd603a593c6fc3c1075b6de1691b414ba041c7fc31"
Jul 17 19:14:20.689620 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:14:20.689382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:14:33.689106 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:33.689070    2563 scope.go:117] "RemoveContainer" containerID="be6bb399c118f4cc79d3ebdd603a593c6fc3c1075b6de1691b414ba041c7fc31"
Jul 17 19:14:34.930663 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:34.930632    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerID="5e0ae7685985a00ec1a79fb8c56a18a86ae688ed98e402f0d4d18c148725f379" exitCode=1
Jul 17 19:14:34.931100 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:34.930708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" event={"ID":"8a1a84cd-b9ba-4005-9642-ae46786aee6d","Type":"ContainerDied","Data":"5e0ae7685985a00ec1a79fb8c56a18a86ae688ed98e402f0d4d18c148725f379"}
Jul 17 19:14:34.931100 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:34.930788    2563 scope.go:117] "RemoveContainer" containerID="be6bb399c118f4cc79d3ebdd603a593c6fc3c1075b6de1691b414ba041c7fc31"
Jul 17 19:14:34.931232 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:34.931156    2563 scope.go:117] "RemoveContainer" containerID="5e0ae7685985a00ec1a79fb8c56a18a86ae688ed98e402f0d4d18c148725f379"
Jul 17 19:14:34.933456 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:14:34.931384    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:14:37.521789 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:37.521753    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:14:37.522223 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:37.521811    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:14:37.522223 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:37.522113    2563 scope.go:117] "RemoveContainer" containerID="5e0ae7685985a00ec1a79fb8c56a18a86ae688ed98e402f0d4d18c148725f379"
Jul 17 19:14:37.522338 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:14:37.522317    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:14:50.689060 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:14:50.689029    2563 scope.go:117] "RemoveContainer" containerID="5e0ae7685985a00ec1a79fb8c56a18a86ae688ed98e402f0d4d18c148725f379"
Jul 17 19:14:50.689506 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:14:50.689228    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:15:02.689060 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:02.688983    2563 scope.go:117] "RemoveContainer" containerID="5e0ae7685985a00ec1a79fb8c56a18a86ae688ed98e402f0d4d18c148725f379"
Jul 17 19:15:02.689447 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:15:02.689200    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:15:14.689062 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:14.689030    2563 scope.go:117] "RemoveContainer" containerID="5e0ae7685985a00ec1a79fb8c56a18a86ae688ed98e402f0d4d18c148725f379"
Jul 17 19:15:16.090806 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:16.090772    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc" exitCode=1
Jul 17 19:15:16.091267 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:16.090824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" event={"ID":"8a1a84cd-b9ba-4005-9642-ae46786aee6d","Type":"ContainerDied","Data":"2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"}
Jul 17 19:15:16.091267 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:16.090873    2563 scope.go:117] "RemoveContainer" containerID="5e0ae7685985a00ec1a79fb8c56a18a86ae688ed98e402f0d4d18c148725f379"
Jul 17 19:15:16.091267 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:16.091204    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:15:16.091410 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:15:16.091389    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:15:17.522664 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:17.522629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:15:17.523054 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:17.522692    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:15:17.523054 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:17.523014    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:15:17.523224 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:15:17.523205    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:15:30.689583 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:30.689546    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:15:30.690088 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:15:30.689728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:15:39.853524 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:39.853488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:15:39.941101 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:39.941059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:15:41.138188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:41.138152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:15:41.238050 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:41.238019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:15:43.689140 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:43.689091    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:15:43.689515 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:15:43.689283    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:15:57.688965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:15:57.688935    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:15:57.689778 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:15:57.689736    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:16:09.688984 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:09.688945    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:16:09.689648 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:09.689190    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:16:22.689779 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:22.689745    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:16:22.690288 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:22.689949    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:16:33.689392 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:33.689297    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:16:33.689965 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:33.689542    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz_odh-ai-gateway-infra(8a1a84cd-b9ba-4005-9642-ae46786aee6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d"
Jul 17 19:16:40.915067 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:40.915034    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"]
Jul 17 19:16:40.915560 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:40.915302    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy" containerID="cri-o://7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec" gracePeriod=30
Jul 17 19:16:40.986516 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:40.986486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:41.087772 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:41.087740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:43.597487 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:43.597442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:16:46.175706 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.175682    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:16:46.183574 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.183510    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"]
Jul 17 19:16:46.193257 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.193218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:46.242726 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-credential-socket\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.242875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt5jz\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-kube-api-access-gt5jz\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.242875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-socket\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.242875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88718a3f-1d10-46c8-91a9-b4431926eb63-istiod-ca-cert\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.242875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-podinfo\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.243088 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-envoy\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.243088 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-certs\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.243088 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242962    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-token\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.243088 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.242989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-data\") pod \"88718a3f-1d10-46c8-91a9-b4431926eb63\" (UID: \"88718a3f-1d10-46c8-91a9-b4431926eb63\") "
Jul 17 19:16:46.243088 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.243014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:16:46.243088 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.243054    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:16:46.243425 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.243382    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.243425 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.243403    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.243524 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.243420    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:16:46.244094 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.243751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88718a3f-1d10-46c8-91a9-b4431926eb63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:16:46.244094 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.243963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-data" (OuterVolumeSpecName: "istio-data") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:16:46.248820 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.248774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:16:46.249079 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.249047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:16:46.249209 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.249138    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-token" (OuterVolumeSpecName: "istio-token") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:16:46.251260 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.251225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-kube-api-access-gt5jz" (OuterVolumeSpecName: "kube-api-access-gt5jz") pod "88718a3f-1d10-46c8-91a9-b4431926eb63" (UID: "88718a3f-1d10-46c8-91a9-b4431926eb63"). InnerVolumeSpecName "kube-api-access-gt5jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:16:46.310014 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.309989    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:16:46.344740 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.344697    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5jz\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-kube-api-access-gt5jz\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.344740 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.344741    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88718a3f-1d10-46c8-91a9-b4431926eb63-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.344935 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.344756    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.344935 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.344769    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.344935 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.344783    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.344935 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.344794    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.344935 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.344806    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88718a3f-1d10-46c8-91a9-b4431926eb63-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.439014 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.438921    2563 generic.go:356] "Generic (PLEG): container finished" podID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerID="7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec" exitCode=0
Jul 17 19:16:46.439014 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.438980    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"
Jul 17 19:16:46.439014 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.438995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" event={"ID":"88718a3f-1d10-46c8-91a9-b4431926eb63","Type":"ContainerDied","Data":"7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec"}
Jul 17 19:16:46.439314 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.439078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr" event={"ID":"88718a3f-1d10-46c8-91a9-b4431926eb63","Type":"ContainerDied","Data":"bd4d2afe215af222970f541ece9051ea2db5a24c4014abf56366b699faea33b7"}
Jul 17 19:16:46.439314 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.439111    2563 scope.go:117] "RemoveContainer" containerID="7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec"
Jul 17 19:16:46.440508 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.440488    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"
Jul 17 19:16:46.440508 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.440497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz" event={"ID":"8a1a84cd-b9ba-4005-9642-ae46786aee6d","Type":"ContainerDied","Data":"51fb8610474114be376adb0ffdaef02194c74a3bdd02cd274cdb6a58a348c5ff"}
Jul 17 19:16:46.445294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.445271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1a84cd-b9ba-4005-9642-ae46786aee6d-openshift-service-ca\") pod \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") "
Jul 17 19:16:46.445418 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.445362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vbvh\" (UniqueName: \"kubernetes.io/projected/8a1a84cd-b9ba-4005-9642-ae46786aee6d-kube-api-access-7vbvh\") pod \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") "
Jul 17 19:16:46.445504 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.445425    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls\") pod \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\" (UID: \"8a1a84cd-b9ba-4005-9642-ae46786aee6d\") "
Jul 17 19:16:46.445714 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.445689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1a84cd-b9ba-4005-9642-ae46786aee6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a1a84cd-b9ba-4005-9642-ae46786aee6d" (UID: "8a1a84cd-b9ba-4005-9642-ae46786aee6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:16:46.447878 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.447851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a1a84cd-b9ba-4005-9642-ae46786aee6d" (UID: "8a1a84cd-b9ba-4005-9642-ae46786aee6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:16:46.447878 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.447866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1a84cd-b9ba-4005-9642-ae46786aee6d-kube-api-access-7vbvh" (OuterVolumeSpecName: "kube-api-access-7vbvh") pod "8a1a84cd-b9ba-4005-9642-ae46786aee6d" (UID: "8a1a84cd-b9ba-4005-9642-ae46786aee6d"). InnerVolumeSpecName "kube-api-access-7vbvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:16:46.448615 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.448600    2563 scope.go:117] "RemoveContainer" containerID="9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc"
Jul 17 19:16:46.456527 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.456501    2563 scope.go:117] "RemoveContainer" containerID="7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec"
Jul 17 19:16:46.456774 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:46.456756    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec\": container with ID starting with 7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec not found: ID does not exist" containerID="7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec"
Jul 17 19:16:46.456825 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.456781    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec"} err="failed to get container status \"7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec\": rpc error: code = NotFound desc = could not find container \"7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec\": container with ID starting with 7b80550ccd785788328f21b650b275bab68f20bea43a592da902098b306844ec not found: ID does not exist"
Jul 17 19:16:46.456825 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.456798    2563 scope.go:117] "RemoveContainer" containerID="9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc"
Jul 17 19:16:46.457033 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:46.457018    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc\": container with ID starting with 9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc not found: ID does not exist" containerID="9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc"
Jul 17 19:16:46.457077 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.457038    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc"} err="failed to get container status \"9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc\": rpc error: code = NotFound desc = could not find container \"9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc\": container with ID starting with 9ddbee09c89ad42313bf31b10bcd8f9576ae3495cbfcc9690521d142c083afbc not found: ID does not exist"
Jul 17 19:16:46.457077 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.457052    2563 scope.go:117] "RemoveContainer" containerID="2e8c55f61c86e2fead965543ddbf87eccc5ce4783a010589c1ed2ef8ff7bcbbc"
Jul 17 19:16:46.459822 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.459802    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"]
Jul 17 19:16:46.463886 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.463863    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b74adac0-openshift-default-795f9744cf-sfknr"]
Jul 17 19:16:46.546506 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.546473    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1a84cd-b9ba-4005-9642-ae46786aee6d-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.546506 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.546501    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1a84cd-b9ba-4005-9642-ae46786aee6d-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.546506 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.546512    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vbvh\" (UniqueName: \"kubernetes.io/projected/8a1a84cd-b9ba-4005-9642-ae46786aee6d-kube-api-access-7vbvh\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:16:46.761732 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.761690    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"]
Jul 17 19:16:46.768114 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:46.768079    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b74adac0-6bc789fc9d-kpcdz"]
Jul 17 19:16:47.383062 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:47.383032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:47.693894 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:47.693811    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" path="/var/lib/kubelet/pods/88718a3f-1d10-46c8-91a9-b4431926eb63/volumes"
Jul 17 19:16:47.694216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:47.694203    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" path="/var/lib/kubelet/pods/8a1a84cd-b9ba-4005-9642-ae46786aee6d/volumes"
Jul 17 19:16:52.643669 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643632    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"]
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643925    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643936    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643944    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643950    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643957    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643963    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643970    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643977    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643982    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.643987    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644002    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644018    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644059    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644066    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:52.644493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644072    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644077    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy"
Jul 17 19:16:52.644493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644082    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644087    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88718a3f-1d10-46c8-91a9-b4431926eb63" containerName="istio-proxy"
Jul 17 19:16:52.644493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644094    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:52.644493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.644099    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:52.648535 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.648517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"
Jul 17 19:16:52.650986 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.650963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:16:52.651111 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.650966    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:16:52.654098 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.654071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"]
Jul 17 19:16:52.799942 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.799901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3ab61-5060-4084-bba9-44f1213df73b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9\" (UID: \"c1f3ab61-5060-4084-bba9-44f1213df73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"
Jul 17 19:16:52.900611 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.900498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3ab61-5060-4084-bba9-44f1213df73b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9\" (UID: \"c1f3ab61-5060-4084-bba9-44f1213df73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"
Jul 17 19:16:52.901177 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.901156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3ab61-5060-4084-bba9-44f1213df73b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9\" (UID: \"c1f3ab61-5060-4084-bba9-44f1213df73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"
Jul 17 19:16:52.961311 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:52.961271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"
Jul 17 19:16:53.093036 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.092928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"]
Jul 17 19:16:53.095992 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:16:53.095954    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f3ab61_5060_4084_bba9_44f1213df73b.slice/crio-2d6a0b4a1aedca812a020ef6f00266f944d8f41078c3ed4c2a2c691662d77cf8 WatchSource:0}: Error finding container 2d6a0b4a1aedca812a020ef6f00266f944d8f41078c3ed4c2a2c691662d77cf8: Status 404 returned error can't find the container with id 2d6a0b4a1aedca812a020ef6f00266f944d8f41078c3ed4c2a2c691662d77cf8
Jul 17 19:16:53.283050 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.283015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:53.384649 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.384612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:53.469931 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.469842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" event={"ID":"c1f3ab61-5060-4084-bba9-44f1213df73b","Type":"ContainerStarted","Data":"5650a5803f206e08a447926bc67886308de76ee5d28a6913eb9f5fbffc39b0ca"}
Jul 17 19:16:53.469931 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.469892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" event={"ID":"c1f3ab61-5060-4084-bba9-44f1213df73b","Type":"ContainerStarted","Data":"2d6a0b4a1aedca812a020ef6f00266f944d8f41078c3ed4c2a2c691662d77cf8"}
Jul 17 19:16:53.486587 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.486536    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" podStartSLOduration=1.486521629 podStartE2EDuration="1.486521629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:16:53.484444643 +0000 UTC m=+2622.395271416" watchObservedRunningTime="2026-07-17 19:16:53.486521629 +0000 UTC m=+2622.397348390"
Jul 17 19:16:53.798484 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.798453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:53.898614 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.898578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:53.995350 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:53.995316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:54.475829 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:54.475798    2563 generic.go:356] "Generic (PLEG): container finished" podID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerID="5650a5803f206e08a447926bc67886308de76ee5d28a6913eb9f5fbffc39b0ca" exitCode=6
Jul 17 19:16:54.476008 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:54.475869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" event={"ID":"c1f3ab61-5060-4084-bba9-44f1213df73b","Type":"ContainerDied","Data":"5650a5803f206e08a447926bc67886308de76ee5d28a6913eb9f5fbffc39b0ca"}
Jul 17 19:16:54.476097 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:54.476083    2563 scope.go:117] "RemoveContainer" containerID="5650a5803f206e08a447926bc67886308de76ee5d28a6913eb9f5fbffc39b0ca"
Jul 17 19:16:55.382842 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.382795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:55.481271 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.481233    2563 generic.go:356] "Generic (PLEG): container finished" podID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerID="c92533011bc0cf124f19984f355cdad21f5ab68bebfd262e7a51da9b3c51142b" exitCode=6
Jul 17 19:16:55.481470 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.481280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" event={"ID":"c1f3ab61-5060-4084-bba9-44f1213df73b","Type":"ContainerDied","Data":"c92533011bc0cf124f19984f355cdad21f5ab68bebfd262e7a51da9b3c51142b"}
Jul 17 19:16:55.481470 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.481363    2563 scope.go:117] "RemoveContainer" containerID="5650a5803f206e08a447926bc67886308de76ee5d28a6913eb9f5fbffc39b0ca"
Jul 17 19:16:55.481592 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.481554    2563 scope.go:117] "RemoveContainer" containerID="c92533011bc0cf124f19984f355cdad21f5ab68bebfd262e7a51da9b3c51142b"
Jul 17 19:16:55.481807 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:55.481789    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9_odh-ai-gateway-infra(c1f3ab61-5060-4084-bba9-44f1213df73b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b"
Jul 17 19:16:55.643609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.643517    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"]
Jul 17 19:16:55.643919 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.643902    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:55.644001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.643922    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:55.644001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.643935    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:55.644001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.643944    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:55.644001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.643957    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:55.644001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.643966    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:55.644271 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.644030    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1a84cd-b9ba-4005-9642-ae46786aee6d" containerName="maas-api"
Jul 17 19:16:55.644271 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.644045    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8573d2b7-6cdb-4051-8846-d77786fa1b46" containerName="revoke-keys"
Jul 17 19:16:55.648139 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.648099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"
Jul 17 19:16:55.653209 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.653181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"]
Jul 17 19:16:55.724700 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.724659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb3305-3496-4523-af39-5632bf0dfd8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt\" (UID: \"43cb3305-3496-4523-af39-5632bf0dfd8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"
Jul 17 19:16:55.825750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.825708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb3305-3496-4523-af39-5632bf0dfd8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt\" (UID: \"43cb3305-3496-4523-af39-5632bf0dfd8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"
Jul 17 19:16:55.826393 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.826368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb3305-3496-4523-af39-5632bf0dfd8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt\" (UID: \"43cb3305-3496-4523-af39-5632bf0dfd8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"
Jul 17 19:16:55.960271 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:55.960176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"
Jul 17 19:16:56.091283 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:56.091253    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"]
Jul 17 19:16:56.094683 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:16:56.094650    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cb3305_3496_4523_af39_5632bf0dfd8d.slice/crio-4092dc9bb4fd7efd0b72ecd3e7ba20e85da42e7f802de2a15ad7b7e7d491b8d7 WatchSource:0}: Error finding container 4092dc9bb4fd7efd0b72ecd3e7ba20e85da42e7f802de2a15ad7b7e7d491b8d7: Status 404 returned error can't find the container with id 4092dc9bb4fd7efd0b72ecd3e7ba20e85da42e7f802de2a15ad7b7e7d491b8d7
Jul 17 19:16:56.487369 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:56.487336    2563 scope.go:117] "RemoveContainer" containerID="c92533011bc0cf124f19984f355cdad21f5ab68bebfd262e7a51da9b3c51142b"
Jul 17 19:16:56.487796 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:56.487570    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9_odh-ai-gateway-infra(c1f3ab61-5060-4084-bba9-44f1213df73b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b"
Jul 17 19:16:56.488501 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:56.488478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" event={"ID":"43cb3305-3496-4523-af39-5632bf0dfd8d","Type":"ContainerStarted","Data":"1149831073cb68066127e609386ef3906fe6cadb91441d95d120d46034030016"}
Jul 17 19:16:56.488631 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:56.488527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" event={"ID":"43cb3305-3496-4523-af39-5632bf0dfd8d","Type":"ContainerStarted","Data":"4092dc9bb4fd7efd0b72ecd3e7ba20e85da42e7f802de2a15ad7b7e7d491b8d7"}
Jul 17 19:16:56.517247 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:56.517190    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" podStartSLOduration=1.5171755230000001 podStartE2EDuration="1.517175523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:16:56.516570155 +0000 UTC m=+2625.427396917" watchObservedRunningTime="2026-07-17 19:16:56.517175523 +0000 UTC m=+2625.428002262"
Jul 17 19:16:57.494288 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:57.494252    2563 generic.go:356] "Generic (PLEG): container finished" podID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerID="1149831073cb68066127e609386ef3906fe6cadb91441d95d120d46034030016" exitCode=6
Jul 17 19:16:57.494687 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:57.494324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" event={"ID":"43cb3305-3496-4523-af39-5632bf0dfd8d","Type":"ContainerDied","Data":"1149831073cb68066127e609386ef3906fe6cadb91441d95d120d46034030016"}
Jul 17 19:16:57.494687 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:57.494496    2563 scope.go:117] "RemoveContainer" containerID="1149831073cb68066127e609386ef3906fe6cadb91441d95d120d46034030016"
Jul 17 19:16:57.613545 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:57.613514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:16:58.499986 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:58.499953    2563 generic.go:356] "Generic (PLEG): container finished" podID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerID="a452077da3082ca591e4fd57c39c241905357d3bbd58973a95b862fe5d25c7c0" exitCode=6
Jul 17 19:16:58.500452 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:58.500008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" event={"ID":"43cb3305-3496-4523-af39-5632bf0dfd8d","Type":"ContainerDied","Data":"a452077da3082ca591e4fd57c39c241905357d3bbd58973a95b862fe5d25c7c0"}
Jul 17 19:16:58.500452 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:58.500061    2563 scope.go:117] "RemoveContainer" containerID="1149831073cb68066127e609386ef3906fe6cadb91441d95d120d46034030016"
Jul 17 19:16:58.500452 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:58.500283    2563 scope.go:117] "RemoveContainer" containerID="a452077da3082ca591e4fd57c39c241905357d3bbd58973a95b862fe5d25c7c0"
Jul 17 19:16:58.500592 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:58.500466    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt_odh-ai-gateway-infra(43cb3305-3496-4523-af39-5632bf0dfd8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d"
Jul 17 19:16:59.505977 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:16:59.505948    2563 scope.go:117] "RemoveContainer" containerID="a452077da3082ca591e4fd57c39c241905357d3bbd58973a95b862fe5d25c7c0"
Jul 17 19:16:59.506397 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:16:59.506139    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt_odh-ai-gateway-infra(43cb3305-3496-4523-af39-5632bf0dfd8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d"
Jul 17 19:17:01.907520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:01.907481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:02.007098 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:02.007063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:02.105467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:02.105438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:02.205759 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:02.205670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:02.571684 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:02.571591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:11.691321 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:11.691287    2563 scope.go:117] "RemoveContainer" containerID="c92533011bc0cf124f19984f355cdad21f5ab68bebfd262e7a51da9b3c51142b"
Jul 17 19:17:12.560655 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:12.560621    2563 generic.go:356] "Generic (PLEG): container finished" podID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerID="2c03f57a75b62240a8df7e1d034b6f6913f15a12ac5e8a6b897fb4583f6a2523" exitCode=6
Jul 17 19:17:12.560845 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:12.560694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" event={"ID":"c1f3ab61-5060-4084-bba9-44f1213df73b","Type":"ContainerDied","Data":"2c03f57a75b62240a8df7e1d034b6f6913f15a12ac5e8a6b897fb4583f6a2523"}
Jul 17 19:17:12.560845 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:12.560756    2563 scope.go:117] "RemoveContainer" containerID="c92533011bc0cf124f19984f355cdad21f5ab68bebfd262e7a51da9b3c51142b"
Jul 17 19:17:12.561163 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:12.561149    2563 scope.go:117] "RemoveContainer" containerID="2c03f57a75b62240a8df7e1d034b6f6913f15a12ac5e8a6b897fb4583f6a2523"
Jul 17 19:17:12.561360 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:12.561337    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9_odh-ai-gateway-infra(c1f3ab61-5060-4084-bba9-44f1213df73b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b"
Jul 17 19:17:13.586139 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:13.586086    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"]
Jul 17 19:17:13.720237 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:13.720210    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"
Jul 17 19:17:13.766503 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:13.766470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3ab61-5060-4084-bba9-44f1213df73b-maas-api-service-ca\") pod \"c1f3ab61-5060-4084-bba9-44f1213df73b\" (UID: \"c1f3ab61-5060-4084-bba9-44f1213df73b\") "
Jul 17 19:17:13.766828 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:13.766806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f3ab61-5060-4084-bba9-44f1213df73b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1f3ab61-5060-4084-bba9-44f1213df73b" (UID: "c1f3ab61-5060-4084-bba9-44f1213df73b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:17:13.867176 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:13.867141    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3ab61-5060-4084-bba9-44f1213df73b-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:17:14.571398 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:14.571366    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"
Jul 17 19:17:14.571578 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:14.571372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9" event={"ID":"c1f3ab61-5060-4084-bba9-44f1213df73b","Type":"ContainerDied","Data":"2d6a0b4a1aedca812a020ef6f00266f944d8f41078c3ed4c2a2c691662d77cf8"}
Jul 17 19:17:14.571578 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:14.571507    2563 scope.go:117] "RemoveContainer" containerID="2c03f57a75b62240a8df7e1d034b6f6913f15a12ac5e8a6b897fb4583f6a2523"
Jul 17 19:17:14.591484 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:14.591455    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"]
Jul 17 19:17:14.596767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:14.596741    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-xhtz9"]
Jul 17 19:17:14.689620 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:14.689585    2563 scope.go:117] "RemoveContainer" containerID="a452077da3082ca591e4fd57c39c241905357d3bbd58973a95b862fe5d25c7c0"
Jul 17 19:17:15.578905 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:15.578817    2563 generic.go:356] "Generic (PLEG): container finished" podID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerID="503c83faea0185924edb638d36b7a51286b56c95865c629344cb484a6a2bf943" exitCode=6
Jul 17 19:17:15.578905 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:15.578865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" event={"ID":"43cb3305-3496-4523-af39-5632bf0dfd8d","Type":"ContainerDied","Data":"503c83faea0185924edb638d36b7a51286b56c95865c629344cb484a6a2bf943"}
Jul 17 19:17:15.579099 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:15.578939    2563 scope.go:117] "RemoveContainer" containerID="a452077da3082ca591e4fd57c39c241905357d3bbd58973a95b862fe5d25c7c0"
Jul 17 19:17:15.579501 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:15.579483    2563 scope.go:117] "RemoveContainer" containerID="503c83faea0185924edb638d36b7a51286b56c95865c629344cb484a6a2bf943"
Jul 17 19:17:15.579695 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:15.579674    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt_odh-ai-gateway-infra(43cb3305-3496-4523-af39-5632bf0dfd8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d"
Jul 17 19:17:15.694221 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:15.694188    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" path="/var/lib/kubelet/pods/c1f3ab61-5060-4084-bba9-44f1213df73b/volumes"
Jul 17 19:17:16.602887 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:16.602851    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"]
Jul 17 19:17:16.737072 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:16.737049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"
Jul 17 19:17:16.790996 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:16.790960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb3305-3496-4523-af39-5632bf0dfd8d-maas-api-service-ca\") pod \"43cb3305-3496-4523-af39-5632bf0dfd8d\" (UID: \"43cb3305-3496-4523-af39-5632bf0dfd8d\") "
Jul 17 19:17:16.791420 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:16.791397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cb3305-3496-4523-af39-5632bf0dfd8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43cb3305-3496-4523-af39-5632bf0dfd8d" (UID: "43cb3305-3496-4523-af39-5632bf0dfd8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:17:16.891814 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:16.891717    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb3305-3496-4523-af39-5632bf0dfd8d-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:17:17.589825 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:17.589800    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"
Jul 17 19:17:17.589974 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:17.589793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt" event={"ID":"43cb3305-3496-4523-af39-5632bf0dfd8d","Type":"ContainerDied","Data":"4092dc9bb4fd7efd0b72ecd3e7ba20e85da42e7f802de2a15ad7b7e7d491b8d7"}
Jul 17 19:17:17.589974 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:17.589930    2563 scope.go:117] "RemoveContainer" containerID="503c83faea0185924edb638d36b7a51286b56c95865c629344cb484a6a2bf943"
Jul 17 19:17:17.610413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:17.610385    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"]
Jul 17 19:17:17.616104 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:17.616084    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-b27pt"]
Jul 17 19:17:17.694107 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:17.694074    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" path="/var/lib/kubelet/pods/43cb3305-3496-4523-af39-5632bf0dfd8d/volumes"
Jul 17 19:17:20.960890 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.960850    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"]
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961174    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961186    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961195    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961201    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961207    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961213    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961232    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961241    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961247    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961289    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961296    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:20.961294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961302    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:20.961675 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.961308    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:20.965288 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.965269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"
Jul 17 19:17:20.967723 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.967701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:17:20.967944 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.967932    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:17:20.970897 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:20.970860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"]
Jul 17 19:17:21.028372 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:21.028328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa7d49-62de-40d9-958d-bcb1c3d6b558-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk\" (UID: \"cfaa7d49-62de-40d9-958d-bcb1c3d6b558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"
Jul 17 19:17:21.128783 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:21.128746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa7d49-62de-40d9-958d-bcb1c3d6b558-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk\" (UID: \"cfaa7d49-62de-40d9-958d-bcb1c3d6b558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"
Jul 17 19:17:21.129406 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:21.129388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa7d49-62de-40d9-958d-bcb1c3d6b558-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk\" (UID: \"cfaa7d49-62de-40d9-958d-bcb1c3d6b558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"
Jul 17 19:17:21.276336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:21.276254    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"
Jul 17 19:17:21.409839 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:21.409806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"]
Jul 17 19:17:21.413339 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:17:21.413312    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaa7d49_62de_40d9_958d_bcb1c3d6b558.slice/crio-7f02e1cea4a4dd52400438d4976f905048b50a09f2a746504de0145c99e695a7 WatchSource:0}: Error finding container 7f02e1cea4a4dd52400438d4976f905048b50a09f2a746504de0145c99e695a7: Status 404 returned error can't find the container with id 7f02e1cea4a4dd52400438d4976f905048b50a09f2a746504de0145c99e695a7
Jul 17 19:17:21.608641 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:21.608608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" event={"ID":"cfaa7d49-62de-40d9-958d-bcb1c3d6b558","Type":"ContainerStarted","Data":"4759efd9300e29fa1715b416fbb276fd1b9c861fe0a70fb00a474e264f16d030"}
Jul 17 19:17:21.608800 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:21.608667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" event={"ID":"cfaa7d49-62de-40d9-958d-bcb1c3d6b558","Type":"ContainerStarted","Data":"7f02e1cea4a4dd52400438d4976f905048b50a09f2a746504de0145c99e695a7"}
Jul 17 19:17:21.624074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:21.624018    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" podStartSLOduration=1.624000063 podStartE2EDuration="1.624000063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:17:21.623082798 +0000 UTC m=+2650.533909558" watchObservedRunningTime="2026-07-17 19:17:21.624000063 +0000 UTC m=+2650.534826824"
Jul 17 19:17:22.614930 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:22.614894    2563 generic.go:356] "Generic (PLEG): container finished" podID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerID="4759efd9300e29fa1715b416fbb276fd1b9c861fe0a70fb00a474e264f16d030" exitCode=6
Jul 17 19:17:22.615431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:22.614985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" event={"ID":"cfaa7d49-62de-40d9-958d-bcb1c3d6b558","Type":"ContainerDied","Data":"4759efd9300e29fa1715b416fbb276fd1b9c861fe0a70fb00a474e264f16d030"}
Jul 17 19:17:22.615431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:22.615191    2563 scope.go:117] "RemoveContainer" containerID="4759efd9300e29fa1715b416fbb276fd1b9c861fe0a70fb00a474e264f16d030"
Jul 17 19:17:23.621587 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.621552    2563 generic.go:356] "Generic (PLEG): container finished" podID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerID="3bbc2be6070311b9cca49a2a8c54020bbfb14b1ffcc4c881e76fcde0da21bcb7" exitCode=6
Jul 17 19:17:23.622008 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.621633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" event={"ID":"cfaa7d49-62de-40d9-958d-bcb1c3d6b558","Type":"ContainerDied","Data":"3bbc2be6070311b9cca49a2a8c54020bbfb14b1ffcc4c881e76fcde0da21bcb7"}
Jul 17 19:17:23.622008 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.621700    2563 scope.go:117] "RemoveContainer" containerID="4759efd9300e29fa1715b416fbb276fd1b9c861fe0a70fb00a474e264f16d030"
Jul 17 19:17:23.622008 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.621910    2563 scope.go:117] "RemoveContainer" containerID="3bbc2be6070311b9cca49a2a8c54020bbfb14b1ffcc4c881e76fcde0da21bcb7"
Jul 17 19:17:23.622169 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:23.622142    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk_odh-ai-gateway-infra(cfaa7d49-62de-40d9-958d-bcb1c3d6b558)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558"
Jul 17 19:17:23.825076 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.825043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"]
Jul 17 19:17:23.825359 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.825346    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:23.825408 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.825361    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:23.825441 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.825421    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f3ab61-5060-4084-bba9-44f1213df73b" containerName="revoke-keys"
Jul 17 19:17:23.825441 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.825429    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cb3305-3496-4523-af39-5632bf0dfd8d" containerName="revoke-keys"
Jul 17 19:17:23.829242 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.829225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"
Jul 17 19:17:23.834960 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.834935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"]
Jul 17 19:17:23.952528 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:23.952428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010b9757-7b7d-49b0-a3c9-215762d7ae6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2\" (UID: \"010b9757-7b7d-49b0-a3c9-215762d7ae6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"
Jul 17 19:17:24.053218 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:24.053180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010b9757-7b7d-49b0-a3c9-215762d7ae6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2\" (UID: \"010b9757-7b7d-49b0-a3c9-215762d7ae6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"
Jul 17 19:17:24.053789 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:24.053769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010b9757-7b7d-49b0-a3c9-215762d7ae6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2\" (UID: \"010b9757-7b7d-49b0-a3c9-215762d7ae6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"
Jul 17 19:17:24.140483 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:24.140441    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"
Jul 17 19:17:24.276332 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:24.276302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"]
Jul 17 19:17:24.278233 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:17:24.278200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010b9757_7b7d_49b0_a3c9_215762d7ae6d.slice/crio-c40923422dea0c4d6d9b5435497d190f2fd08cb451eddbed043415cdf3314e52 WatchSource:0}: Error finding container c40923422dea0c4d6d9b5435497d190f2fd08cb451eddbed043415cdf3314e52: Status 404 returned error can't find the container with id c40923422dea0c4d6d9b5435497d190f2fd08cb451eddbed043415cdf3314e52
Jul 17 19:17:24.627494 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:24.627465    2563 scope.go:117] "RemoveContainer" containerID="3bbc2be6070311b9cca49a2a8c54020bbfb14b1ffcc4c881e76fcde0da21bcb7"
Jul 17 19:17:24.627921 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:24.627695    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk_odh-ai-gateway-infra(cfaa7d49-62de-40d9-958d-bcb1c3d6b558)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558"
Jul 17 19:17:24.628670 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:24.628645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" event={"ID":"010b9757-7b7d-49b0-a3c9-215762d7ae6d","Type":"ContainerStarted","Data":"a545f77dd981ad6efdda83375212dd24e298d2c9c91dca4ca7de5e6d3bdbd557"}
Jul 17 19:17:24.628796 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:24.628688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" event={"ID":"010b9757-7b7d-49b0-a3c9-215762d7ae6d","Type":"ContainerStarted","Data":"c40923422dea0c4d6d9b5435497d190f2fd08cb451eddbed043415cdf3314e52"}
Jul 17 19:17:24.657722 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:24.657669    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" podStartSLOduration=1.657652659 podStartE2EDuration="1.657652659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:17:24.655956169 +0000 UTC m=+2653.566782929" watchObservedRunningTime="2026-07-17 19:17:24.657652659 +0000 UTC m=+2653.568479420"
Jul 17 19:17:25.634041 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:25.634000    2563 generic.go:356] "Generic (PLEG): container finished" podID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerID="a545f77dd981ad6efdda83375212dd24e298d2c9c91dca4ca7de5e6d3bdbd557" exitCode=6
Jul 17 19:17:25.634479 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:25.634081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" event={"ID":"010b9757-7b7d-49b0-a3c9-215762d7ae6d","Type":"ContainerDied","Data":"a545f77dd981ad6efdda83375212dd24e298d2c9c91dca4ca7de5e6d3bdbd557"}
Jul 17 19:17:25.634479 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:25.634246    2563 scope.go:117] "RemoveContainer" containerID="a545f77dd981ad6efdda83375212dd24e298d2c9c91dca4ca7de5e6d3bdbd557"
Jul 17 19:17:26.642224 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:26.642190    2563 generic.go:356] "Generic (PLEG): container finished" podID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerID="2eaaf95f9b051811e5897ac8c772fa8f2c55c465b9293720f53435075187aa2b" exitCode=6
Jul 17 19:17:26.642674 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:26.642279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" event={"ID":"010b9757-7b7d-49b0-a3c9-215762d7ae6d","Type":"ContainerDied","Data":"2eaaf95f9b051811e5897ac8c772fa8f2c55c465b9293720f53435075187aa2b"}
Jul 17 19:17:26.642674 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:26.642334    2563 scope.go:117] "RemoveContainer" containerID="a545f77dd981ad6efdda83375212dd24e298d2c9c91dca4ca7de5e6d3bdbd557"
Jul 17 19:17:26.642674 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:26.642575    2563 scope.go:117] "RemoveContainer" containerID="2eaaf95f9b051811e5897ac8c772fa8f2c55c465b9293720f53435075187aa2b"
Jul 17 19:17:26.642829 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:26.642807    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2_odh-ai-gateway-infra(010b9757-7b7d-49b0-a3c9-215762d7ae6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d"
Jul 17 19:17:27.648530 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:27.648490    2563 scope.go:117] "RemoveContainer" containerID="2eaaf95f9b051811e5897ac8c772fa8f2c55c465b9293720f53435075187aa2b"
Jul 17 19:17:27.648907 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:27.648731    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2_odh-ai-gateway-infra(010b9757-7b7d-49b0-a3c9-215762d7ae6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d"
Jul 17 19:17:28.505094 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:28.505040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:34.384082 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:34.384037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:34.437342 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:34.437312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:34.507618 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:34.507581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:34.609271 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:34.609240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:37.689410 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:37.689379    2563 scope.go:117] "RemoveContainer" containerID="3bbc2be6070311b9cca49a2a8c54020bbfb14b1ffcc4c881e76fcde0da21bcb7"
Jul 17 19:17:38.689680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:38.689595    2563 scope.go:117] "RemoveContainer" containerID="2eaaf95f9b051811e5897ac8c772fa8f2c55c465b9293720f53435075187aa2b"
Jul 17 19:17:38.696640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:38.696612    2563 generic.go:356] "Generic (PLEG): container finished" podID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerID="709461cdf22ac06e8f43c9b5c37efdb7cd9fbb8afd4a9b8d6a062fa6ab9e4c44" exitCode=6
Jul 17 19:17:38.696741 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:38.696680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" event={"ID":"cfaa7d49-62de-40d9-958d-bcb1c3d6b558","Type":"ContainerDied","Data":"709461cdf22ac06e8f43c9b5c37efdb7cd9fbb8afd4a9b8d6a062fa6ab9e4c44"}
Jul 17 19:17:38.696741 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:38.696736    2563 scope.go:117] "RemoveContainer" containerID="3bbc2be6070311b9cca49a2a8c54020bbfb14b1ffcc4c881e76fcde0da21bcb7"
Jul 17 19:17:38.696982 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:38.696968    2563 scope.go:117] "RemoveContainer" containerID="709461cdf22ac06e8f43c9b5c37efdb7cd9fbb8afd4a9b8d6a062fa6ab9e4c44"
Jul 17 19:17:38.697197 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:38.697175    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk_odh-ai-gateway-infra(cfaa7d49-62de-40d9-958d-bcb1c3d6b558)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558"
Jul 17 19:17:39.703216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:39.703184    2563 generic.go:356] "Generic (PLEG): container finished" podID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerID="28ec8fe75743dd23b3129757fb900bc52226c16d9c472d0f9e879997df42d081" exitCode=6
Jul 17 19:17:39.703644 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:39.703262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" event={"ID":"010b9757-7b7d-49b0-a3c9-215762d7ae6d","Type":"ContainerDied","Data":"28ec8fe75743dd23b3129757fb900bc52226c16d9c472d0f9e879997df42d081"}
Jul 17 19:17:39.703644 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:39.703332    2563 scope.go:117] "RemoveContainer" containerID="2eaaf95f9b051811e5897ac8c772fa8f2c55c465b9293720f53435075187aa2b"
Jul 17 19:17:39.703644 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:39.703593    2563 scope.go:117] "RemoveContainer" containerID="28ec8fe75743dd23b3129757fb900bc52226c16d9c472d0f9e879997df42d081"
Jul 17 19:17:39.703838 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:39.703814    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2_odh-ai-gateway-infra(010b9757-7b7d-49b0-a3c9-215762d7ae6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d"
Jul 17 19:17:39.723535 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:39.723509    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"]
Jul 17 19:17:39.856147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:39.856109    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"
Jul 17 19:17:39.992061 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:39.991958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa7d49-62de-40d9-958d-bcb1c3d6b558-maas-api-service-ca\") pod \"cfaa7d49-62de-40d9-958d-bcb1c3d6b558\" (UID: \"cfaa7d49-62de-40d9-958d-bcb1c3d6b558\") "
Jul 17 19:17:39.992331 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:39.992308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaa7d49-62de-40d9-958d-bcb1c3d6b558-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfaa7d49-62de-40d9-958d-bcb1c3d6b558" (UID: "cfaa7d49-62de-40d9-958d-bcb1c3d6b558"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:17:40.093411 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:40.093364    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa7d49-62de-40d9-958d-bcb1c3d6b558-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:17:40.708333 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:40.708299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk" event={"ID":"cfaa7d49-62de-40d9-958d-bcb1c3d6b558","Type":"ContainerDied","Data":"7f02e1cea4a4dd52400438d4976f905048b50a09f2a746504de0145c99e695a7"}
Jul 17 19:17:40.708333 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:40.708325    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"
Jul 17 19:17:40.708899 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:40.708362    2563 scope.go:117] "RemoveContainer" containerID="709461cdf22ac06e8f43c9b5c37efdb7cd9fbb8afd4a9b8d6a062fa6ab9e4c44"
Jul 17 19:17:40.729437 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:40.729405    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"]
Jul 17 19:17:40.732239 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:40.732215    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"]
Jul 17 19:17:40.737687 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:40.737659    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-nkzqk"]
Jul 17 19:17:40.858302 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:40.858275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"
Jul 17 19:17:41.001302 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.001209    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010b9757-7b7d-49b0-a3c9-215762d7ae6d-maas-api-service-ca\") pod \"010b9757-7b7d-49b0-a3c9-215762d7ae6d\" (UID: \"010b9757-7b7d-49b0-a3c9-215762d7ae6d\") "
Jul 17 19:17:41.001580 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.001555    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010b9757-7b7d-49b0-a3c9-215762d7ae6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "010b9757-7b7d-49b0-a3c9-215762d7ae6d" (UID: "010b9757-7b7d-49b0-a3c9-215762d7ae6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:17:41.102053 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.102007    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010b9757-7b7d-49b0-a3c9-215762d7ae6d-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:17:41.693784 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.693752    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" path="/var/lib/kubelet/pods/cfaa7d49-62de-40d9-958d-bcb1c3d6b558/volumes"
Jul 17 19:17:41.716513 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.716469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2" event={"ID":"010b9757-7b7d-49b0-a3c9-215762d7ae6d","Type":"ContainerDied","Data":"c40923422dea0c4d6d9b5435497d190f2fd08cb451eddbed043415cdf3314e52"}
Jul 17 19:17:41.716904 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.716535    2563 scope.go:117] "RemoveContainer" containerID="28ec8fe75743dd23b3129757fb900bc52226c16d9c472d0f9e879997df42d081"
Jul 17 19:17:41.716904 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.716485    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"
Jul 17 19:17:41.731370 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.731338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"]
Jul 17 19:17:41.735167 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:41.735144    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6scx2"]
Jul 17 19:17:43.693935 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:43.693901    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" path="/var/lib/kubelet/pods/010b9757-7b7d-49b0-a3c9-215762d7ae6d/volumes"
Jul 17 19:17:50.447778 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:50.447747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:52.687730 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.687691    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"]
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688039    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688051    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688060    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688066    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688072    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688078    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688085    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688090    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688159    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688165    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688172    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688178    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:52.688179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.688183    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:52.691188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.691167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"
Jul 17 19:17:52.693773 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.693745    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:17:52.693773 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.693760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:17:52.697299 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.697273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"]
Jul 17 19:17:52.805602 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.805563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397a92-a36e-41f6-a81a-b6f557ca3bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd\" (UID: \"b4397a92-a36e-41f6-a81a-b6f557ca3bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"
Jul 17 19:17:52.906504 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.906462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397a92-a36e-41f6-a81a-b6f557ca3bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd\" (UID: \"b4397a92-a36e-41f6-a81a-b6f557ca3bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"
Jul 17 19:17:52.907098 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:52.907076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397a92-a36e-41f6-a81a-b6f557ca3bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd\" (UID: \"b4397a92-a36e-41f6-a81a-b6f557ca3bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"
Jul 17 19:17:53.006951 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:53.006853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"
Jul 17 19:17:53.138934 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:53.138812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"]
Jul 17 19:17:53.141734 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:17:53.141709    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4397a92_a36e_41f6_a81a_b6f557ca3bed.slice/crio-1a69c812f39d14f6e779b7f4e42654d801b75e9e49fd4693ad90197e24902950 WatchSource:0}: Error finding container 1a69c812f39d14f6e779b7f4e42654d801b75e9e49fd4693ad90197e24902950: Status 404 returned error can't find the container with id 1a69c812f39d14f6e779b7f4e42654d801b75e9e49fd4693ad90197e24902950
Jul 17 19:17:53.768170 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:53.768051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" event={"ID":"b4397a92-a36e-41f6-a81a-b6f557ca3bed","Type":"ContainerStarted","Data":"127911a6b887d112bbf498ad7b8c833bbccbe9e21753f4c2b1b0a98d6036f044"}
Jul 17 19:17:53.768170 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:53.768113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" event={"ID":"b4397a92-a36e-41f6-a81a-b6f557ca3bed","Type":"ContainerStarted","Data":"1a69c812f39d14f6e779b7f4e42654d801b75e9e49fd4693ad90197e24902950"}
Jul 17 19:17:53.784532 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:53.784476    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" podStartSLOduration=1.784456371 podStartE2EDuration="1.784456371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:17:53.783057094 +0000 UTC m=+2682.693883891" watchObservedRunningTime="2026-07-17 19:17:53.784456371 +0000 UTC m=+2682.695283109"
Jul 17 19:17:54.773835 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:54.773798    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerID="127911a6b887d112bbf498ad7b8c833bbccbe9e21753f4c2b1b0a98d6036f044" exitCode=6
Jul 17 19:17:54.774263 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:54.773863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" event={"ID":"b4397a92-a36e-41f6-a81a-b6f557ca3bed","Type":"ContainerDied","Data":"127911a6b887d112bbf498ad7b8c833bbccbe9e21753f4c2b1b0a98d6036f044"}
Jul 17 19:17:54.774263 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:54.774013    2563 scope.go:117] "RemoveContainer" containerID="127911a6b887d112bbf498ad7b8c833bbccbe9e21753f4c2b1b0a98d6036f044"
Jul 17 19:17:55.682829 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.682748    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"]
Jul 17 19:17:55.683036 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.683024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:55.683084 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.683037    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="010b9757-7b7d-49b0-a3c9-215762d7ae6d" containerName="revoke-keys"
Jul 17 19:17:55.683084 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.683049    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:55.683084 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.683054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:55.683204 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.683131    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfaa7d49-62de-40d9-958d-bcb1c3d6b558" containerName="revoke-keys"
Jul 17 19:17:55.685837 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.685820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"
Jul 17 19:17:55.695270 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.695242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"]
Jul 17 19:17:55.780903 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.780869    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerID="d2c8d21d7671f7970739220a4d0bfed24dbab2856b1156e8b2a9514541035657" exitCode=6
Jul 17 19:17:55.781338 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.780919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" event={"ID":"b4397a92-a36e-41f6-a81a-b6f557ca3bed","Type":"ContainerDied","Data":"d2c8d21d7671f7970739220a4d0bfed24dbab2856b1156e8b2a9514541035657"}
Jul 17 19:17:55.781338 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.780980    2563 scope.go:117] "RemoveContainer" containerID="127911a6b887d112bbf498ad7b8c833bbccbe9e21753f4c2b1b0a98d6036f044"
Jul 17 19:17:55.781338 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.781231    2563 scope.go:117] "RemoveContainer" containerID="d2c8d21d7671f7970739220a4d0bfed24dbab2856b1156e8b2a9514541035657"
Jul 17 19:17:55.781496 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:55.781473    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd_odh-ai-gateway-infra(b4397a92-a36e-41f6-a81a-b6f557ca3bed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed"
Jul 17 19:17:55.832514 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.832469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8791a1-c082-4fe4-90ae-ec991bef4ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv\" (UID: \"1b8791a1-c082-4fe4-90ae-ec991bef4ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"
Jul 17 19:17:55.933544 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.933454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8791a1-c082-4fe4-90ae-ec991bef4ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv\" (UID: \"1b8791a1-c082-4fe4-90ae-ec991bef4ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"
Jul 17 19:17:55.934078 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.934059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8791a1-c082-4fe4-90ae-ec991bef4ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv\" (UID: \"1b8791a1-c082-4fe4-90ae-ec991bef4ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"
Jul 17 19:17:55.998259 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:55.998221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"
Jul 17 19:17:56.137013 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:56.136841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"]
Jul 17 19:17:56.140045 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:17:56.139999    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8791a1_c082_4fe4_90ae_ec991bef4ebf.slice/crio-1ad1601be7c6fc88844b1daa4aea7ac755663b6bd7eb9cd2a7569c31c3bc4d40 WatchSource:0}: Error finding container 1ad1601be7c6fc88844b1daa4aea7ac755663b6bd7eb9cd2a7569c31c3bc4d40: Status 404 returned error can't find the container with id 1ad1601be7c6fc88844b1daa4aea7ac755663b6bd7eb9cd2a7569c31c3bc4d40
Jul 17 19:17:56.179821 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:56.179789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:56.288628 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:56.288556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:56.379878 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:56.379845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:17:56.788252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:56.788226    2563 scope.go:117] "RemoveContainer" containerID="d2c8d21d7671f7970739220a4d0bfed24dbab2856b1156e8b2a9514541035657"
Jul 17 19:17:56.788612 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:56.788490    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd_odh-ai-gateway-infra(b4397a92-a36e-41f6-a81a-b6f557ca3bed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed"
Jul 17 19:17:56.789966 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:56.789943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" event={"ID":"1b8791a1-c082-4fe4-90ae-ec991bef4ebf","Type":"ContainerStarted","Data":"8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2"}
Jul 17 19:17:56.790088 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:56.789984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" event={"ID":"1b8791a1-c082-4fe4-90ae-ec991bef4ebf","Type":"ContainerStarted","Data":"1ad1601be7c6fc88844b1daa4aea7ac755663b6bd7eb9cd2a7569c31c3bc4d40"}
Jul 17 19:17:56.790146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:56.790108    2563 scope.go:117] "RemoveContainer" containerID="8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2"
Jul 17 19:17:57.795877 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:57.795791    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerID="8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2" exitCode=6
Jul 17 19:17:57.795877 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:57.795815    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerID="fe0dc8363aef7fe7c880d4041e889ecdab825642f463d2e12ef9d75d645e2d1a" exitCode=6
Jul 17 19:17:57.796326 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:57.795880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" event={"ID":"1b8791a1-c082-4fe4-90ae-ec991bef4ebf","Type":"ContainerDied","Data":"8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2"}
Jul 17 19:17:57.796326 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:57.795939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" event={"ID":"1b8791a1-c082-4fe4-90ae-ec991bef4ebf","Type":"ContainerDied","Data":"fe0dc8363aef7fe7c880d4041e889ecdab825642f463d2e12ef9d75d645e2d1a"}
Jul 17 19:17:57.796326 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:57.795966    2563 scope.go:117] "RemoveContainer" containerID="8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2"
Jul 17 19:17:57.796326 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:57.796062    2563 scope.go:117] "RemoveContainer" containerID="fe0dc8363aef7fe7c880d4041e889ecdab825642f463d2e12ef9d75d645e2d1a"
Jul 17 19:17:57.796326 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:57.796282    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv_odh-ai-gateway-infra(1b8791a1-c082-4fe4-90ae-ec991bef4ebf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf"
Jul 17 19:17:57.806464 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:57.806441    2563 scope.go:117] "RemoveContainer" containerID="8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2"
Jul 17 19:17:57.806769 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:57.806748    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2\": container with ID starting with 8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2 not found: ID does not exist" containerID="8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2"
Jul 17 19:17:57.806853 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:57.806779    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2"} err="failed to get container status \"8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2\": rpc error: code = NotFound desc = could not find container \"8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2\": container with ID starting with 8ca84a8c2af5571de7a1ecada77b914fe522d679039e9f4a8089a2f91c1fb7c2 not found: ID does not exist"
Jul 17 19:17:58.802033 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:17:58.802004    2563 scope.go:117] "RemoveContainer" containerID="fe0dc8363aef7fe7c880d4041e889ecdab825642f463d2e12ef9d75d645e2d1a"
Jul 17 19:17:58.802451 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:17:58.802223    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv_odh-ai-gateway-infra(1b8791a1-c082-4fe4-90ae-ec991bef4ebf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf"
Jul 17 19:18:06.908222 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:06.908107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:07.680699 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:07.680661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:08.469023 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:08.468984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:08.688868 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:08.688838    2563 scope.go:117] "RemoveContainer" containerID="d2c8d21d7671f7970739220a4d0bfed24dbab2856b1156e8b2a9514541035657"
Jul 17 19:18:09.847717 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:09.847687    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerID="544c2dc80483806d7516057b5e59b86196a035382c6df85e5e43ae8b18ca0ed8" exitCode=6
Jul 17 19:18:09.848114 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:09.847758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" event={"ID":"b4397a92-a36e-41f6-a81a-b6f557ca3bed","Type":"ContainerDied","Data":"544c2dc80483806d7516057b5e59b86196a035382c6df85e5e43ae8b18ca0ed8"}
Jul 17 19:18:09.848114 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:09.847820    2563 scope.go:117] "RemoveContainer" containerID="d2c8d21d7671f7970739220a4d0bfed24dbab2856b1156e8b2a9514541035657"
Jul 17 19:18:09.848114 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:09.848086    2563 scope.go:117] "RemoveContainer" containerID="544c2dc80483806d7516057b5e59b86196a035382c6df85e5e43ae8b18ca0ed8"
Jul 17 19:18:09.848347 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:09.848301    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd_odh-ai-gateway-infra(b4397a92-a36e-41f6-a81a-b6f557ca3bed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed"
Jul 17 19:18:10.875050 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:10.875021    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"]
Jul 17 19:18:11.004175 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004152    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"
Jul 17 19:18:11.004399 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004381    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"]
Jul 17 19:18:11.004710 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004694    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.004756 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004713    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.004756 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004724    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.004756 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004730    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.004756 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.004756 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.004907 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004789    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.004907 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004799    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.004907 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.004805    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" containerName="revoke-keys"
Jul 17 19:18:11.013085 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.013057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"
Jul 17 19:18:11.014679 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.014616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"]
Jul 17 19:18:11.056887 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.056851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892df402-1824-44c6-9c3e-7bd910d7c761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7\" (UID: \"892df402-1824-44c6-9c3e-7bd910d7c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"
Jul 17 19:18:11.158099 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.158006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397a92-a36e-41f6-a81a-b6f557ca3bed-maas-api-service-ca\") pod \"b4397a92-a36e-41f6-a81a-b6f557ca3bed\" (UID: \"b4397a92-a36e-41f6-a81a-b6f557ca3bed\") "
Jul 17 19:18:11.158280 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.158216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892df402-1824-44c6-9c3e-7bd910d7c761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7\" (UID: \"892df402-1824-44c6-9c3e-7bd910d7c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"
Jul 17 19:18:11.158443 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.158422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4397a92-a36e-41f6-a81a-b6f557ca3bed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4397a92-a36e-41f6-a81a-b6f557ca3bed" (UID: "b4397a92-a36e-41f6-a81a-b6f557ca3bed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:18:11.158829 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.158810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892df402-1824-44c6-9c3e-7bd910d7c761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7\" (UID: \"892df402-1824-44c6-9c3e-7bd910d7c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"
Jul 17 19:18:11.258626 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.258594    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4397a92-a36e-41f6-a81a-b6f557ca3bed-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:18:11.327570 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.327534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"
Jul 17 19:18:11.455111 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.455080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"]
Jul 17 19:18:11.457507 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:18:11.457482    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892df402_1824_44c6_9c3e_7bd910d7c761.slice/crio-42c99842f2cf4dc0fbba2feef749b29422f38513a54e1ec860ed5a9c2de32293 WatchSource:0}: Error finding container 42c99842f2cf4dc0fbba2feef749b29422f38513a54e1ec860ed5a9c2de32293: Status 404 returned error can't find the container with id 42c99842f2cf4dc0fbba2feef749b29422f38513a54e1ec860ed5a9c2de32293
Jul 17 19:18:11.702667 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.702644    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:18:11.707150 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.707064    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:18:11.859448 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.859422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"
Jul 17 19:18:11.859610 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.859462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd" event={"ID":"b4397a92-a36e-41f6-a81a-b6f557ca3bed","Type":"ContainerDied","Data":"1a69c812f39d14f6e779b7f4e42654d801b75e9e49fd4693ad90197e24902950"}
Jul 17 19:18:11.859610 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.859526    2563 scope.go:117] "RemoveContainer" containerID="544c2dc80483806d7516057b5e59b86196a035382c6df85e5e43ae8b18ca0ed8"
Jul 17 19:18:11.860956 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.860924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" event={"ID":"892df402-1824-44c6-9c3e-7bd910d7c761","Type":"ContainerStarted","Data":"3559265776f860c0ec94b29a866fff3c56c8b4e17cde42049a45d7d36f1a8118"}
Jul 17 19:18:11.861050 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.860980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" event={"ID":"892df402-1824-44c6-9c3e-7bd910d7c761","Type":"ContainerStarted","Data":"42c99842f2cf4dc0fbba2feef749b29422f38513a54e1ec860ed5a9c2de32293"}
Jul 17 19:18:11.875316 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.875290    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"]
Jul 17 19:18:11.879282 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.879259    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-ghrdd"]
Jul 17 19:18:11.893799 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:11.893755    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" podStartSLOduration=1.893737773 podStartE2EDuration="1.893737773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:18:11.893191047 +0000 UTC m=+2700.804017809" watchObservedRunningTime="2026-07-17 19:18:11.893737773 +0000 UTC m=+2700.804564534"
Jul 17 19:18:12.867636 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:12.867600    2563 generic.go:356] "Generic (PLEG): container finished" podID="892df402-1824-44c6-9c3e-7bd910d7c761" containerID="3559265776f860c0ec94b29a866fff3c56c8b4e17cde42049a45d7d36f1a8118" exitCode=6
Jul 17 19:18:12.867855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:12.867649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" event={"ID":"892df402-1824-44c6-9c3e-7bd910d7c761","Type":"ContainerDied","Data":"3559265776f860c0ec94b29a866fff3c56c8b4e17cde42049a45d7d36f1a8118"}
Jul 17 19:18:12.867918 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:12.867903    2563 scope.go:117] "RemoveContainer" containerID="3559265776f860c0ec94b29a866fff3c56c8b4e17cde42049a45d7d36f1a8118"
Jul 17 19:18:13.549175 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.549113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:13.663931 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.663898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:13.689392 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.689364    2563 scope.go:117] "RemoveContainer" containerID="fe0dc8363aef7fe7c880d4041e889ecdab825642f463d2e12ef9d75d645e2d1a"
Jul 17 19:18:13.693983 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.693957    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4397a92-a36e-41f6-a81a-b6f557ca3bed" path="/var/lib/kubelet/pods/b4397a92-a36e-41f6-a81a-b6f557ca3bed/volumes"
Jul 17 19:18:13.749173 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.749077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:13.851722 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.851690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:13.869728 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.869696    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"]
Jul 17 19:18:13.873767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.873741    2563 generic.go:356] "Generic (PLEG): container finished" podID="892df402-1824-44c6-9c3e-7bd910d7c761" containerID="49cba260e96da3a1f95d12a159d58f058fb128bbfff05db6acd12ba6813f047f" exitCode=6
Jul 17 19:18:13.874067 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.874053    2563 scope.go:117] "RemoveContainer" containerID="49cba260e96da3a1f95d12a159d58f058fb128bbfff05db6acd12ba6813f047f"
Jul 17 19:18:13.874279 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:13.874261    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7_odh-ai-gateway-infra(892df402-1824-44c6-9c3e-7bd910d7c761)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" podUID="892df402-1824-44c6-9c3e-7bd910d7c761"
Jul 17 19:18:13.874383 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.874364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" event={"ID":"892df402-1824-44c6-9c3e-7bd910d7c761","Type":"ContainerDied","Data":"49cba260e96da3a1f95d12a159d58f058fb128bbfff05db6acd12ba6813f047f"}
Jul 17 19:18:13.874418 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.874404    2563 scope.go:117] "RemoveContainer" containerID="3559265776f860c0ec94b29a866fff3c56c8b4e17cde42049a45d7d36f1a8118"
Jul 17 19:18:13.874486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.874466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"
Jul 17 19:18:13.876616 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.876593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad90245b-3a32-4583-8770-b6f1667cd644-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn\" (UID: \"ad90245b-3a32-4583-8770-b6f1667cd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"
Jul 17 19:18:13.882740 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.882713    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"]
Jul 17 19:18:13.977244 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.977216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad90245b-3a32-4583-8770-b6f1667cd644-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn\" (UID: \"ad90245b-3a32-4583-8770-b6f1667cd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"
Jul 17 19:18:13.977766 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:13.977747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad90245b-3a32-4583-8770-b6f1667cd644-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn\" (UID: \"ad90245b-3a32-4583-8770-b6f1667cd644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"
Jul 17 19:18:14.186397 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.186359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"
Jul 17 19:18:14.317596 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.317460    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"]
Jul 17 19:18:14.320308 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:18:14.320276    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad90245b_3a32_4583_8770_b6f1667cd644.slice/crio-307d098b7df4b85c144513569d960839eac7cac92cfefb534a579159abb57f36 WatchSource:0}: Error finding container 307d098b7df4b85c144513569d960839eac7cac92cfefb534a579159abb57f36: Status 404 returned error can't find the container with id 307d098b7df4b85c144513569d960839eac7cac92cfefb534a579159abb57f36
Jul 17 19:18:14.604342 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.604307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:14.708340 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.708300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:14.806008 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.805974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:14.880133 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.880093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" event={"ID":"ad90245b-3a32-4583-8770-b6f1667cd644","Type":"ContainerStarted","Data":"4b3d9e2b227a72b8ec65dfb15b535d1d1f3bd787e49a00a4d6b615441060812b"}
Jul 17 19:18:14.880322 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.880174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" event={"ID":"ad90245b-3a32-4583-8770-b6f1667cd644","Type":"ContainerStarted","Data":"307d098b7df4b85c144513569d960839eac7cac92cfefb534a579159abb57f36"}
Jul 17 19:18:14.881956 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.881931    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerID="280279294a8595bec5b2b1e9b997f59f6d96df2c040263bc23e5fc0a0295d087" exitCode=6
Jul 17 19:18:14.882074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.882012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" event={"ID":"1b8791a1-c082-4fe4-90ae-ec991bef4ebf","Type":"ContainerDied","Data":"280279294a8595bec5b2b1e9b997f59f6d96df2c040263bc23e5fc0a0295d087"}
Jul 17 19:18:14.882154 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.882070    2563 scope.go:117] "RemoveContainer" containerID="fe0dc8363aef7fe7c880d4041e889ecdab825642f463d2e12ef9d75d645e2d1a"
Jul 17 19:18:14.882344 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.882327    2563 scope.go:117] "RemoveContainer" containerID="280279294a8595bec5b2b1e9b997f59f6d96df2c040263bc23e5fc0a0295d087"
Jul 17 19:18:14.882531 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:14.882509    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv_odh-ai-gateway-infra(1b8791a1-c082-4fe4-90ae-ec991bef4ebf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf"
Jul 17 19:18:14.883880 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.883865    2563 scope.go:117] "RemoveContainer" containerID="49cba260e96da3a1f95d12a159d58f058fb128bbfff05db6acd12ba6813f047f"
Jul 17 19:18:14.884020 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:14.884003    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7_odh-ai-gateway-infra(892df402-1824-44c6-9c3e-7bd910d7c761)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" podUID="892df402-1824-44c6-9c3e-7bd910d7c761"
Jul 17 19:18:14.901910 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:14.901865    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" podStartSLOduration=1.901848902 podStartE2EDuration="1.901848902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:18:14.900375257 +0000 UTC m=+2703.811202018" watchObservedRunningTime="2026-07-17 19:18:14.901848902 +0000 UTC m=+2703.812675665"
Jul 17 19:18:15.888722 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:15.888685    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad90245b-3a32-4583-8770-b6f1667cd644" containerID="4b3d9e2b227a72b8ec65dfb15b535d1d1f3bd787e49a00a4d6b615441060812b" exitCode=6
Jul 17 19:18:15.889200 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:15.888773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" event={"ID":"ad90245b-3a32-4583-8770-b6f1667cd644","Type":"ContainerDied","Data":"4b3d9e2b227a72b8ec65dfb15b535d1d1f3bd787e49a00a4d6b615441060812b"}
Jul 17 19:18:15.889200 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:15.888967    2563 scope.go:117] "RemoveContainer" containerID="4b3d9e2b227a72b8ec65dfb15b535d1d1f3bd787e49a00a4d6b615441060812b"
Jul 17 19:18:15.925157 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:15.925109    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"]
Jul 17 19:18:16.056835 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.056807    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"
Jul 17 19:18:16.092206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.092180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8791a1-c082-4fe4-90ae-ec991bef4ebf-maas-api-service-ca\") pod \"1b8791a1-c082-4fe4-90ae-ec991bef4ebf\" (UID: \"1b8791a1-c082-4fe4-90ae-ec991bef4ebf\") "
Jul 17 19:18:16.092515 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.092483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8791a1-c082-4fe4-90ae-ec991bef4ebf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b8791a1-c082-4fe4-90ae-ec991bef4ebf" (UID: "1b8791a1-c082-4fe4-90ae-ec991bef4ebf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:18:16.193169 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.193142    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8791a1-c082-4fe4-90ae-ec991bef4ebf-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:18:16.324700 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.324656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:16.897017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.896981    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad90245b-3a32-4583-8770-b6f1667cd644" containerID="819b34a0c3c6aa126728b67a49b44d82b0206e72615c50bb878ee268749276bc" exitCode=6
Jul 17 19:18:16.897507 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.897073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" event={"ID":"ad90245b-3a32-4583-8770-b6f1667cd644","Type":"ContainerDied","Data":"819b34a0c3c6aa126728b67a49b44d82b0206e72615c50bb878ee268749276bc"}
Jul 17 19:18:16.897507 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.897163    2563 scope.go:117] "RemoveContainer" containerID="4b3d9e2b227a72b8ec65dfb15b535d1d1f3bd787e49a00a4d6b615441060812b"
Jul 17 19:18:16.897507 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.897262    2563 scope.go:117] "RemoveContainer" containerID="819b34a0c3c6aa126728b67a49b44d82b0206e72615c50bb878ee268749276bc"
Jul 17 19:18:16.897677 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:16.897506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn_odh-ai-gateway-infra(ad90245b-3a32-4583-8770-b6f1667cd644)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" podUID="ad90245b-3a32-4583-8770-b6f1667cd644"
Jul 17 19:18:16.898561 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.898542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv" event={"ID":"1b8791a1-c082-4fe4-90ae-ec991bef4ebf","Type":"ContainerDied","Data":"1ad1601be7c6fc88844b1daa4aea7ac755663b6bd7eb9cd2a7569c31c3bc4d40"}
Jul 17 19:18:16.898652 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.898561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"
Jul 17 19:18:16.908667 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.908648    2563 scope.go:117] "RemoveContainer" containerID="280279294a8595bec5b2b1e9b997f59f6d96df2c040263bc23e5fc0a0295d087"
Jul 17 19:18:16.925726 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.925698    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"]
Jul 17 19:18:16.929472 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:16.929449    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7snhv"]
Jul 17 19:18:17.699305 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:17.697361    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" path="/var/lib/kubelet/pods/1b8791a1-c082-4fe4-90ae-ec991bef4ebf/volumes"
Jul 17 19:18:17.850940 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:17.850903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:18:17.905401 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:17.905363    2563 scope.go:117] "RemoveContainer" containerID="819b34a0c3c6aa126728b67a49b44d82b0206e72615c50bb878ee268749276bc"
Jul 17 19:18:17.905888 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:17.905572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn_odh-ai-gateway-infra(ad90245b-3a32-4583-8770-b6f1667cd644)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" podUID="ad90245b-3a32-4583-8770-b6f1667cd644"
Jul 17 19:18:22.427273 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:22.427239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:22.492636 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:22.492600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:23.096454 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:23.096424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:26.705098 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:26.705060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:18:28.689557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:28.689524    2563 scope.go:117] "RemoveContainer" containerID="49cba260e96da3a1f95d12a159d58f058fb128bbfff05db6acd12ba6813f047f"
Jul 17 19:18:29.523583 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:29.523544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:29.643287 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:29.643254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:29.813374 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:29.813346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:29.957046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:29.956954    2563 generic.go:356] "Generic (PLEG): container finished" podID="892df402-1824-44c6-9c3e-7bd910d7c761" containerID="6e779509053fb737d3186071afdc7fc722c8e9608e6290f568069bc77df1af47" exitCode=6
Jul 17 19:18:29.957046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:29.957014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" event={"ID":"892df402-1824-44c6-9c3e-7bd910d7c761","Type":"ContainerDied","Data":"6e779509053fb737d3186071afdc7fc722c8e9608e6290f568069bc77df1af47"}
Jul 17 19:18:29.957273 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:29.957074    2563 scope.go:117] "RemoveContainer" containerID="49cba260e96da3a1f95d12a159d58f058fb128bbfff05db6acd12ba6813f047f"
Jul 17 19:18:29.957405 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:29.957378    2563 scope.go:117] "RemoveContainer" containerID="6e779509053fb737d3186071afdc7fc722c8e9608e6290f568069bc77df1af47"
Jul 17 19:18:29.957598 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:29.957575    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7_odh-ai-gateway-infra(892df402-1824-44c6-9c3e-7bd910d7c761)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" podUID="892df402-1824-44c6-9c3e-7bd910d7c761"
Jul 17 19:18:30.066946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.066908    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"]
Jul 17 19:18:30.067231 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.067216    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:30.067231 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.067231    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:30.067231 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.067242    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:30.067379 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.067249    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:30.067379 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.067265    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:30.067379 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.067270    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:30.067379 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.067321    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:30.067379 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.067331    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:30.071271 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.071251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.073851 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.073831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bd0f3bb1-openshift-default-dockercfg-qwrjf\""
Jul 17 19:18:30.082604 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.082576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"]
Jul 17 19:18:30.096774 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.096739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:30.107179 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-socket\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.107327 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5qn\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-kube-api-access-dd5qn\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.107327 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827a5779-ed1a-4f71-b4da-9231c9f7962c-istiod-ca-cert\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.107327 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-certs\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.107436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-credential-socket\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.107436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-podinfo\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.107436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-envoy\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.107436 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-token\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.107565 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.107452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-data\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.208450 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-certs\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.208450 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-credential-socket\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.208680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-podinfo\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.208680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-envoy\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.208680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-token\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.208680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-data\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.208680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-socket\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.208680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5qn\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-kube-api-access-dd5qn\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.209032 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827a5779-ed1a-4f71-b4da-9231c9f7962c-istiod-ca-cert\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.209032 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.208833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-certs\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.209181 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.209046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-data\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.209181 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.209098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-credential-socket\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.209454 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.209228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-socket\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.209454 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.209376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827a5779-ed1a-4f71-b4da-9231c9f7962c-istiod-ca-cert\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.210459 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.210432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:30.211007 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.210991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-envoy\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.211373 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.211353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-podinfo\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.219657 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.219633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-token\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.220686 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.220659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5qn\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-kube-api-access-dd5qn\") pod \"e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") " pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.381499 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.381458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:30.517272 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.517249    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"]
Jul 17 19:18:30.519476 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:18:30.519452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827a5779_ed1a_4f71_b4da_9231c9f7962c.slice/crio-64dcbb87f77fde6d6289c9d0ac24102ecf2140799e59e8ad18c9ad5eec8f4ece WatchSource:0}: Error finding container 64dcbb87f77fde6d6289c9d0ac24102ecf2140799e59e8ad18c9ad5eec8f4ece: Status 404 returned error can't find the container with id 64dcbb87f77fde6d6289c9d0ac24102ecf2140799e59e8ad18c9ad5eec8f4ece
Jul 17 19:18:30.521531 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.521496    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:18:30.521599 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.521564    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:18:30.521633 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.521618    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:18:30.689129 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.689097    2563 scope.go:117] "RemoveContainer" containerID="819b34a0c3c6aa126728b67a49b44d82b0206e72615c50bb878ee268749276bc"
Jul 17 19:18:30.964617 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.964580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" event={"ID":"ad90245b-3a32-4583-8770-b6f1667cd644","Type":"ContainerStarted","Data":"00ad64526f595853315a69c7427d460eaf5f5a595c9e2e060534d355e62fd698"}
Jul 17 19:18:30.966045 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.966020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" event={"ID":"827a5779-ed1a-4f71-b4da-9231c9f7962c","Type":"ContainerStarted","Data":"221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54"}
Jul 17 19:18:30.966183 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.966070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" event={"ID":"827a5779-ed1a-4f71-b4da-9231c9f7962c","Type":"ContainerStarted","Data":"64dcbb87f77fde6d6289c9d0ac24102ecf2140799e59e8ad18c9ad5eec8f4ece"}
Jul 17 19:18:30.985589 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.985509    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"]
Jul 17 19:18:30.999409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:30.999356    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" podStartSLOduration=0.999335677 podStartE2EDuration="999.335677ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:18:30.998599506 +0000 UTC m=+2719.909426295" watchObservedRunningTime="2026-07-17 19:18:30.999335677 +0000 UTC m=+2719.910162440"
Jul 17 19:18:31.102029 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.102002    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"
Jul 17 19:18:31.217623 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.217593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892df402-1824-44c6-9c3e-7bd910d7c761-maas-api-service-ca\") pod \"892df402-1824-44c6-9c3e-7bd910d7c761\" (UID: \"892df402-1824-44c6-9c3e-7bd910d7c761\") "
Jul 17 19:18:31.218028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.218000    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892df402-1824-44c6-9c3e-7bd910d7c761-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "892df402-1824-44c6-9c3e-7bd910d7c761" (UID: "892df402-1824-44c6-9c3e-7bd910d7c761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:18:31.319202 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.319170    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/892df402-1824-44c6-9c3e-7bd910d7c761-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:18:31.381823 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.381789    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:31.387194 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.387167    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:31.971573 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.971480    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"
Jul 17 19:18:31.971573 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.971496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7" event={"ID":"892df402-1824-44c6-9c3e-7bd910d7c761","Type":"ContainerDied","Data":"42c99842f2cf4dc0fbba2feef749b29422f38513a54e1ec860ed5a9c2de32293"}
Jul 17 19:18:31.971573 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.971567    2563 scope.go:117] "RemoveContainer" containerID="6e779509053fb737d3186071afdc7fc722c8e9608e6290f568069bc77df1af47"
Jul 17 19:18:31.973566 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.973521    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad90245b-3a32-4583-8770-b6f1667cd644" containerID="00ad64526f595853315a69c7427d460eaf5f5a595c9e2e060534d355e62fd698" exitCode=6
Jul 17 19:18:31.973695 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.973556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" event={"ID":"ad90245b-3a32-4583-8770-b6f1667cd644","Type":"ContainerDied","Data":"00ad64526f595853315a69c7427d460eaf5f5a595c9e2e060534d355e62fd698"}
Jul 17 19:18:31.973920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.973902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:31.974025 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.974010    2563 scope.go:117] "RemoveContainer" containerID="00ad64526f595853315a69c7427d460eaf5f5a595c9e2e060534d355e62fd698"
Jul 17 19:18:31.974270 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:31.974245    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn_odh-ai-gateway-infra(ad90245b-3a32-4583-8770-b6f1667cd644)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" podUID="ad90245b-3a32-4583-8770-b6f1667cd644"
Jul 17 19:18:31.975056 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.975032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:31.980027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.980012    2563 scope.go:117] "RemoveContainer" containerID="819b34a0c3c6aa126728b67a49b44d82b0206e72615c50bb878ee268749276bc"
Jul 17 19:18:31.988203 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.988171    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"]
Jul 17 19:18:31.990574 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.990554    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"]
Jul 17 19:18:31.992635 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:31.992612    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-ffxt7"]
Jul 17 19:18:32.892854 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.892824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:32.997668 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.997635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:32.999083 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999054    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"]
Jul 17 19:18:32.999560 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999540    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:32.999643 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999563    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:32.999643 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999576    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:32.999643 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:32.999643 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999598    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:32.999643 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999606    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:32.999934 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999680    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:32.999934 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999692    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8791a1-c082-4fe4-90ae-ec991bef4ebf" containerName="revoke-keys"
Jul 17 19:18:32.999934 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:32.999701    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:33.004197 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.004173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.006884 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.006860    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bd0f3bb1-serving-cert\""
Jul 17 19:18:33.007017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.006894    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:18:33.007017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.007002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:18:33.014882 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.014856    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"]
Jul 17 19:18:33.035107 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.035071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.035368 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.035343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e77385-d21a-45ec-ba8e-2274dcde417b-openshift-service-ca\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.035459 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.035397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4fs\" (UniqueName: \"kubernetes.io/projected/a3e77385-d21a-45ec-ba8e-2274dcde417b-kube-api-access-dm4fs\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.093369 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.093328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:33.135464 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.135423    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"
Jul 17 19:18:33.135892 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.135873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.135946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.135929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e77385-d21a-45ec-ba8e-2274dcde417b-openshift-service-ca\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.135987 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.135960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4fs\" (UniqueName: \"kubernetes.io/projected/a3e77385-d21a-45ec-ba8e-2274dcde417b-kube-api-access-dm4fs\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.136029 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:33.136003    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-serving-cert: secret "maas-api-e2e-mt-bd0f3bb1-serving-cert" not found
Jul 17 19:18:33.136078 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:33.136066    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls podName:a3e77385-d21a-45ec-ba8e-2274dcde417b nodeName:}" failed. No retries permitted until 2026-07-17 19:18:33.636047281 +0000 UTC m=+2722.546874026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls") pod "maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq" (UID: "a3e77385-d21a-45ec-ba8e-2274dcde417b") : secret "maas-api-e2e-mt-bd0f3bb1-serving-cert" not found
Jul 17 19:18:33.136749 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.136723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e77385-d21a-45ec-ba8e-2274dcde417b-openshift-service-ca\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.144880 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.144809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4fs\" (UniqueName: \"kubernetes.io/projected/a3e77385-d21a-45ec-ba8e-2274dcde417b-kube-api-access-dm4fs\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.236885 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.236845    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad90245b-3a32-4583-8770-b6f1667cd644-maas-api-service-ca\") pod \"ad90245b-3a32-4583-8770-b6f1667cd644\" (UID: \"ad90245b-3a32-4583-8770-b6f1667cd644\") "
Jul 17 19:18:33.237287 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.237266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad90245b-3a32-4583-8770-b6f1667cd644-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad90245b-3a32-4583-8770-b6f1667cd644" (UID: "ad90245b-3a32-4583-8770-b6f1667cd644"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:18:33.314576 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.314542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:33.338021 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.337991    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad90245b-3a32-4583-8770-b6f1667cd644-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:18:33.538069 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.537974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:33.639597 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.639569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:33.640695 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.640671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.643171 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.643153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls\") pod \"maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.694064 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.693861    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" path="/var/lib/kubelet/pods/892df402-1824-44c6-9c3e-7bd910d7c761/volumes"
Jul 17 19:18:33.917263 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.917225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:33.987968 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.987939    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"
Jul 17 19:18:33.987968 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.987957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn" event={"ID":"ad90245b-3a32-4583-8770-b6f1667cd644","Type":"ContainerDied","Data":"307d098b7df4b85c144513569d960839eac7cac92cfefb534a579159abb57f36"}
Jul 17 19:18:33.988189 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:33.988039    2563 scope.go:117] "RemoveContainer" containerID="00ad64526f595853315a69c7427d460eaf5f5a595c9e2e060534d355e62fd698"
Jul 17 19:18:34.002598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:34.002569    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"]
Jul 17 19:18:34.008802 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:34.008773    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hbmzn"]
Jul 17 19:18:34.054076 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:34.054041    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"]
Jul 17 19:18:34.056898 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:18:34.056876    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e77385_d21a_45ec_ba8e_2274dcde417b.slice/crio-5e926b6fff5edffe6ea75e12162db20ff395259b6637257ffdcc7bcf600bda57 WatchSource:0}: Error finding container 5e926b6fff5edffe6ea75e12162db20ff395259b6637257ffdcc7bcf600bda57: Status 404 returned error can't find the container with id 5e926b6fff5edffe6ea75e12162db20ff395259b6637257ffdcc7bcf600bda57
Jul 17 19:18:34.058295 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:34.058274    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:18:35.000216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:35.000182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq" event={"ID":"a3e77385-d21a-45ec-ba8e-2274dcde417b","Type":"ContainerStarted","Data":"68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e"}
Jul 17 19:18:35.000438 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:35.000369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:35.000438 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:35.000413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq" event={"ID":"a3e77385-d21a-45ec-ba8e-2274dcde417b","Type":"ContainerStarted","Data":"5e926b6fff5edffe6ea75e12162db20ff395259b6637257ffdcc7bcf600bda57"}
Jul 17 19:18:35.017670 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:35.017618    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq" podStartSLOduration=2.641064776 podStartE2EDuration="3.017602285s" totalImagesPullingTime="376.537509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:18:35.016609413 +0000 UTC m=+2723.927436213" watchObservedRunningTime="2026-07-17 19:18:35.017602285 +0000 UTC m=+2723.928429045"
Jul 17 19:18:35.496491 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:35.496450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:18:35.693853 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:35.693818    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" path="/var/lib/kubelet/pods/ad90245b-3a32-4583-8770-b6f1667cd644/volumes"
Jul 17 19:18:37.387606 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:37.387555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:38.037757 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:38.037725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:38.136335 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:38.136302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:38.476217 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:38.476185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:41.011006 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:41.010931    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:44.038995 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:44.038961    2563 generic.go:356] "Generic (PLEG): container finished" podID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerID="221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54" exitCode=0
Jul 17 19:18:44.039458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:44.039035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" event={"ID":"827a5779-ed1a-4f71-b4da-9231c9f7962c","Type":"ContainerDied","Data":"221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54"}
Jul 17 19:18:44.039458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:44.039414    2563 scope.go:117] "RemoveContainer" containerID="221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54"
Jul 17 19:18:44.040413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:44.040375    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:18:44.040524 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:44.040453    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:18:44.040524 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:44.040503    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:18:44.612415 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:44.612386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:44.710181 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:44.710150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:45.045090 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:45.045004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" event={"ID":"827a5779-ed1a-4f71-b4da-9231c9f7962c","Type":"ContainerStarted","Data":"5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981"}
Jul 17 19:18:45.382587 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:45.382549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:45.383915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:45.383884    2563 patch_prober.go:28] interesting pod/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 17 19:18:45.384420 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:45.384388    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 17 19:18:46.382160 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:46.382096    2563 patch_prober.go:28] interesting pod/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 17 19:18:46.382889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:46.382851    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 17 19:18:46.974807 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:46.974759    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:47.382762 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:47.382291    2563 patch_prober.go:28] interesting pod/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 17 19:18:47.382762 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:47.382356    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 17 19:18:48.382308 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:48.382262    2563 patch_prober.go:28] interesting pod/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 17 19:18:48.382531 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:48.382333    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 17 19:18:49.385872 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:49.385840    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:49.386887 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:49.386869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:18:53.892188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:53.892159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:53.991737 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:53.991702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:54.995466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:54.995426    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"]
Jul 17 19:18:54.995932 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:54.995727    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq" podUID="a3e77385-d21a-45ec-ba8e-2274dcde417b" containerName="maas-api" containerID="cri-o://68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e" gracePeriod=30
Jul 17 19:18:55.020603 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.020577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:18:55.252355 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.252288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:55.341394 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.341357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls\") pod \"a3e77385-d21a-45ec-ba8e-2274dcde417b\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") "
Jul 17 19:18:55.341580 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.341408    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm4fs\" (UniqueName: \"kubernetes.io/projected/a3e77385-d21a-45ec-ba8e-2274dcde417b-kube-api-access-dm4fs\") pod \"a3e77385-d21a-45ec-ba8e-2274dcde417b\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") "
Jul 17 19:18:55.341580 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.341447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e77385-d21a-45ec-ba8e-2274dcde417b-openshift-service-ca\") pod \"a3e77385-d21a-45ec-ba8e-2274dcde417b\" (UID: \"a3e77385-d21a-45ec-ba8e-2274dcde417b\") "
Jul 17 19:18:55.341870 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.341845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e77385-d21a-45ec-ba8e-2274dcde417b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3e77385-d21a-45ec-ba8e-2274dcde417b" (UID: "a3e77385-d21a-45ec-ba8e-2274dcde417b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:18:55.343664 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.343644    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3e77385-d21a-45ec-ba8e-2274dcde417b" (UID: "a3e77385-d21a-45ec-ba8e-2274dcde417b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:18:55.343723 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.343688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e77385-d21a-45ec-ba8e-2274dcde417b-kube-api-access-dm4fs" (OuterVolumeSpecName: "kube-api-access-dm4fs") pod "a3e77385-d21a-45ec-ba8e-2274dcde417b" (UID: "a3e77385-d21a-45ec-ba8e-2274dcde417b"). InnerVolumeSpecName "kube-api-access-dm4fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:18:55.442656 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.442617    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3e77385-d21a-45ec-ba8e-2274dcde417b-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:18:55.442656 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.442649    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4fs\" (UniqueName: \"kubernetes.io/projected/a3e77385-d21a-45ec-ba8e-2274dcde417b-kube-api-access-dm4fs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:18:55.442656 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.442660    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3e77385-d21a-45ec-ba8e-2274dcde417b-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:18:55.724351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724320    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"]
Jul 17 19:18:55.724630 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724617    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:18:55.724678 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724632    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:18:55.724678 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724640    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:18:55.724678 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724645    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:18:55.724678 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724657    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e77385-d21a-45ec-ba8e-2274dcde417b" containerName="maas-api"
Jul 17 19:18:55.724678 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724663    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e77385-d21a-45ec-ba8e-2274dcde417b" containerName="maas-api"
Jul 17 19:18:55.724831 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724706    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:18:55.724831 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724715    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:18:55.724831 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724721    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e77385-d21a-45ec-ba8e-2274dcde417b" containerName="maas-api"
Jul 17 19:18:55.724831 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.724729    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="892df402-1824-44c6-9c3e-7bd910d7c761" containerName="revoke-keys"
Jul 17 19:18:55.727821 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.727806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"
Jul 17 19:18:55.730041 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.730024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:18:55.734877 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.734849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"]
Jul 17 19:18:55.846475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.846437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de884e27-f8a8-4bfc-89cd-7f587e2189f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29\" (UID: \"de884e27-f8a8-4bfc-89cd-7f587e2189f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"
Jul 17 19:18:55.947437 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.947399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de884e27-f8a8-4bfc-89cd-7f587e2189f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29\" (UID: \"de884e27-f8a8-4bfc-89cd-7f587e2189f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"
Jul 17 19:18:55.947987 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:55.947967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de884e27-f8a8-4bfc-89cd-7f587e2189f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29\" (UID: \"de884e27-f8a8-4bfc-89cd-7f587e2189f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"
Jul 17 19:18:56.038420 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.038315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"
Jul 17 19:18:56.092768 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.092710    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3e77385-d21a-45ec-ba8e-2274dcde417b" containerID="68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e" exitCode=0
Jul 17 19:18:56.092915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.092805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq" event={"ID":"a3e77385-d21a-45ec-ba8e-2274dcde417b","Type":"ContainerDied","Data":"68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e"}
Jul 17 19:18:56.092915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.092863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq" event={"ID":"a3e77385-d21a-45ec-ba8e-2274dcde417b","Type":"ContainerDied","Data":"5e926b6fff5edffe6ea75e12162db20ff395259b6637257ffdcc7bcf600bda57"}
Jul 17 19:18:56.092915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.092896    2563 scope.go:117] "RemoveContainer" containerID="68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e"
Jul 17 19:18:56.093083 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.093065    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"
Jul 17 19:18:56.104063 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.104040    2563 scope.go:117] "RemoveContainer" containerID="68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e"
Jul 17 19:18:56.104547 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:56.104510    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e\": container with ID starting with 68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e not found: ID does not exist" containerID="68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e"
Jul 17 19:18:56.104645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.104557    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e"} err="failed to get container status \"68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e\": rpc error: code = NotFound desc = could not find container \"68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e\": container with ID starting with 68d983a949ada01ee6db2cae824e039f1084682a2e8dc59577aeb434ea6bba3e not found: ID does not exist"
Jul 17 19:18:56.110142 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.110087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"]
Jul 17 19:18:56.113803 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.113780    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd0f3bb1-b6f8c9b7b-jp5sq"]
Jul 17 19:18:56.173291 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:56.173259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"]
Jul 17 19:18:56.177720 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:18:56.177693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d WatchSource:0}: Error finding container 1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d: Status 404 returned error can't find the container with id 1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d
Jul 17 19:18:57.099446 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:57.099409    2563 generic.go:356] "Generic (PLEG): container finished" podID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerID="8787863d6c2d2482d41a29fe00a9f312be8858baa778764808b24d45d61a5b0f" exitCode=6
Jul 17 19:18:57.099446 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:57.099446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" event={"ID":"de884e27-f8a8-4bfc-89cd-7f587e2189f7","Type":"ContainerDied","Data":"8787863d6c2d2482d41a29fe00a9f312be8858baa778764808b24d45d61a5b0f"}
Jul 17 19:18:57.099853 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:57.099481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" event={"ID":"de884e27-f8a8-4bfc-89cd-7f587e2189f7","Type":"ContainerStarted","Data":"1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d"}
Jul 17 19:18:57.099853 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:57.099655    2563 scope.go:117] "RemoveContainer" containerID="8787863d6c2d2482d41a29fe00a9f312be8858baa778764808b24d45d61a5b0f"
Jul 17 19:18:57.693394 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:57.693364    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e77385-d21a-45ec-ba8e-2274dcde417b" path="/var/lib/kubelet/pods/a3e77385-d21a-45ec-ba8e-2274dcde417b/volumes"
Jul 17 19:18:57.998558 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:57.998469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:18:58.110996 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:58.110965    2563 generic.go:356] "Generic (PLEG): container finished" podID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerID="7d7a7a08ae7359f550e03df744d2819ae0fd07a2b5b6f01498eb478f536c083f" exitCode=6
Jul 17 19:18:58.111456 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:58.111044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" event={"ID":"de884e27-f8a8-4bfc-89cd-7f587e2189f7","Type":"ContainerDied","Data":"7d7a7a08ae7359f550e03df744d2819ae0fd07a2b5b6f01498eb478f536c083f"}
Jul 17 19:18:58.111456 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:58.111162    2563 scope.go:117] "RemoveContainer" containerID="8787863d6c2d2482d41a29fe00a9f312be8858baa778764808b24d45d61a5b0f"
Jul 17 19:18:58.111456 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:58.111380    2563 scope.go:117] "RemoveContainer" containerID="7d7a7a08ae7359f550e03df744d2819ae0fd07a2b5b6f01498eb478f536c083f"
Jul 17 19:18:58.111658 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:58.111615    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29_odh-ai-gateway-infra(de884e27-f8a8-4bfc-89cd-7f587e2189f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7"
Jul 17 19:18:59.116799 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:18:59.116764    2563 scope.go:117] "RemoveContainer" containerID="7d7a7a08ae7359f550e03df744d2819ae0fd07a2b5b6f01498eb478f536c083f"
Jul 17 19:18:59.117235 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:18:59.116942    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29_odh-ai-gateway-infra(de884e27-f8a8-4bfc-89cd-7f587e2189f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7"
Jul 17 19:19:00.035880 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:00.035845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:06.183483 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:06.183429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:06.287129 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:06.287085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:10.687550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:10.687497    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"]
Jul 17 19:19:10.689299 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:10.689262    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy" containerID="cri-o://5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981" gracePeriod=30
Jul 17 19:19:10.708889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:10.708839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:10.923471 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:10.923439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:11.050069 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:11.049983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:11.143735 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:11.143697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:11.287919 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:11.287890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:12.689198 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:12.689168    2563 scope.go:117] "RemoveContainer" containerID="7d7a7a08ae7359f550e03df744d2819ae0fd07a2b5b6f01498eb478f536c083f"
Jul 17 19:19:13.175711 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:13.175674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" event={"ID":"de884e27-f8a8-4bfc-89cd-7f587e2189f7","Type":"ContainerStarted","Data":"b82635e78501e1358afe68b58c2096d7f507ea05a8a5bab5f138220ee3bf65ee"}
Jul 17 19:19:13.194621 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:13.194569    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" podStartSLOduration=18.19455383 podStartE2EDuration="18.19455383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:19:13.193299034 +0000 UTC m=+2762.104125821" watchObservedRunningTime="2026-07-17 19:19:13.19455383 +0000 UTC m=+2762.105380590"
Jul 17 19:19:13.920597 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:13.920560    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"]
Jul 17 19:19:13.920982 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:13.920855    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:19:13.920982 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:13.920865    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:19:13.920982 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:13.920915    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad90245b-3a32-4583-8770-b6f1667cd644" containerName="revoke-keys"
Jul 17 19:19:13.923631 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:13.923615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"
Jul 17 19:19:13.934035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:13.934007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"]
Jul 17 19:19:14.000563 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.000526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2613d6de-4e70-400e-b27f-13c3396b996a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2\" (UID: \"2613d6de-4e70-400e-b27f-13c3396b996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"
Jul 17 19:19:14.101576 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.101539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2613d6de-4e70-400e-b27f-13c3396b996a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2\" (UID: \"2613d6de-4e70-400e-b27f-13c3396b996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"
Jul 17 19:19:14.102193 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.102172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2613d6de-4e70-400e-b27f-13c3396b996a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2\" (UID: \"2613d6de-4e70-400e-b27f-13c3396b996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"
Jul 17 19:19:14.182623 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.182529    2563 generic.go:356] "Generic (PLEG): container finished" podID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerID="b82635e78501e1358afe68b58c2096d7f507ea05a8a5bab5f138220ee3bf65ee" exitCode=6
Jul 17 19:19:14.182623 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.182599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" event={"ID":"de884e27-f8a8-4bfc-89cd-7f587e2189f7","Type":"ContainerDied","Data":"b82635e78501e1358afe68b58c2096d7f507ea05a8a5bab5f138220ee3bf65ee"}
Jul 17 19:19:14.182859 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.182663    2563 scope.go:117] "RemoveContainer" containerID="7d7a7a08ae7359f550e03df744d2819ae0fd07a2b5b6f01498eb478f536c083f"
Jul 17 19:19:14.182976 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.182954    2563 scope.go:117] "RemoveContainer" containerID="b82635e78501e1358afe68b58c2096d7f507ea05a8a5bab5f138220ee3bf65ee"
Jul 17 19:19:14.183236 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:14.183215    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29_odh-ai-gateway-infra(de884e27-f8a8-4bfc-89cd-7f587e2189f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7"
Jul 17 19:19:14.210437 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.210410    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"]
Jul 17 19:19:14.233822 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.233796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"
Jul 17 19:19:14.370742 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:14.370696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"]
Jul 17 19:19:14.373365 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:19:14.373336    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2613d6de_4e70_400e_b27f_13c3396b996a.slice/crio-27ea0d4ea9885d6735165d31da7c0a59e8bb70f3de4a459026f5558e41fc3a1a WatchSource:0}: Error finding container 27ea0d4ea9885d6735165d31da7c0a59e8bb70f3de4a459026f5558e41fc3a1a: Status 404 returned error can't find the container with id 27ea0d4ea9885d6735165d31da7c0a59e8bb70f3de4a459026f5558e41fc3a1a
Jul 17 19:19:15.189763 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.189727    2563 generic.go:356] "Generic (PLEG): container finished" podID="2613d6de-4e70-400e-b27f-13c3396b996a" containerID="1dfc337eb31ca4f5c6df4309a857fa40d45091fac70ba7854633fbcdd7da9b51" exitCode=6
Jul 17 19:19:15.190167 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.189772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" event={"ID":"2613d6de-4e70-400e-b27f-13c3396b996a","Type":"ContainerDied","Data":"1dfc337eb31ca4f5c6df4309a857fa40d45091fac70ba7854633fbcdd7da9b51"}
Jul 17 19:19:15.190167 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.189821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" event={"ID":"2613d6de-4e70-400e-b27f-13c3396b996a","Type":"ContainerStarted","Data":"27ea0d4ea9885d6735165d31da7c0a59e8bb70f3de4a459026f5558e41fc3a1a"}
Jul 17 19:19:15.190167 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.190012    2563 scope.go:117] "RemoveContainer" containerID="1dfc337eb31ca4f5c6df4309a857fa40d45091fac70ba7854633fbcdd7da9b51"
Jul 17 19:19:15.326231 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.326199    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"
Jul 17 19:19:15.412301 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.412257    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de884e27-f8a8-4bfc-89cd-7f587e2189f7-maas-api-service-ca\") pod \"de884e27-f8a8-4bfc-89cd-7f587e2189f7\" (UID: \"de884e27-f8a8-4bfc-89cd-7f587e2189f7\") "
Jul 17 19:19:15.412667 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.412641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de884e27-f8a8-4bfc-89cd-7f587e2189f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de884e27-f8a8-4bfc-89cd-7f587e2189f7" (UID: "de884e27-f8a8-4bfc-89cd-7f587e2189f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:19:15.513306 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.513232    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de884e27-f8a8-4bfc-89cd-7f587e2189f7-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:15.938825 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:15.938803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:19:16.017602 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-socket\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.017602 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017611    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-certs\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.017861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017641    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827a5779-ed1a-4f71-b4da-9231c9f7962c-istiod-ca-cert\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.017861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-envoy\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.017861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017690    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-data\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.017861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-credential-socket\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.017861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd5qn\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-kube-api-access-dd5qn\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.017861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-token\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.017861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-podinfo\") pod \"827a5779-ed1a-4f71-b4da-9231c9f7962c\" (UID: \"827a5779-ed1a-4f71-b4da-9231c9f7962c\") "
Jul 17 19:19:16.018252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017921    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:19:16.018252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017947    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:19:16.018252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.017961    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:19:16.018252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.018147    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.018252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.018167    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.018252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.018182    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.018483 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.018365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827a5779-ed1a-4f71-b4da-9231c9f7962c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:19:16.018814 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.018790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-data" (OuterVolumeSpecName: "istio-data") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:19:16.020074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.020041    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-kube-api-access-dd5qn" (OuterVolumeSpecName: "kube-api-access-dd5qn") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "kube-api-access-dd5qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:19:16.020074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.020054    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-token" (OuterVolumeSpecName: "istio-token") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:19:16.020454 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.020438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:19:16.020517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.020479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "827a5779-ed1a-4f71-b4da-9231c9f7962c" (UID: "827a5779-ed1a-4f71-b4da-9231c9f7962c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:19:16.049657 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:16.049570    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:16.049786 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:16.049690    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:16.119311 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.119277    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/827a5779-ed1a-4f71-b4da-9231c9f7962c-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.119311 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.119306    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.119311 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.119314    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.119550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.119344    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd5qn\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-kube-api-access-dd5qn\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.119550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.119355    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.119550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.119363    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/827a5779-ed1a-4f71-b4da-9231c9f7962c-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.196111 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.196072    2563 generic.go:356] "Generic (PLEG): container finished" podID="2613d6de-4e70-400e-b27f-13c3396b996a" containerID="8a99446849f5b4fc0741e79b4ebb5cdca76dce0225a22dd6ea26e859f5d49bdf" exitCode=6
Jul 17 19:19:16.196609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.196175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" event={"ID":"2613d6de-4e70-400e-b27f-13c3396b996a","Type":"ContainerDied","Data":"8a99446849f5b4fc0741e79b4ebb5cdca76dce0225a22dd6ea26e859f5d49bdf"}
Jul 17 19:19:16.196609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.196285    2563 scope.go:117] "RemoveContainer" containerID="1dfc337eb31ca4f5c6df4309a857fa40d45091fac70ba7854633fbcdd7da9b51"
Jul 17 19:19:16.196609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.196443    2563 scope.go:117] "RemoveContainer" containerID="8a99446849f5b4fc0741e79b4ebb5cdca76dce0225a22dd6ea26e859f5d49bdf"
Jul 17 19:19:16.196758 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:16.196686    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2_odh-ai-gateway-infra(2613d6de-4e70-400e-b27f-13c3396b996a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" podUID="2613d6de-4e70-400e-b27f-13c3396b996a"
Jul 17 19:19:16.197607 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.197485    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"
Jul 17 19:19:16.197607 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.197507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29" event={"ID":"de884e27-f8a8-4bfc-89cd-7f587e2189f7","Type":"ContainerDied","Data":"1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d"}
Jul 17 19:19:16.199233 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.199213    2563 generic.go:356] "Generic (PLEG): container finished" podID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerID="5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981" exitCode=0
Jul 17 19:19:16.199328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.199247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" event={"ID":"827a5779-ed1a-4f71-b4da-9231c9f7962c","Type":"ContainerDied","Data":"5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981"}
Jul 17 19:19:16.199328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.199286    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"
Jul 17 19:19:16.199328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.199312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh" event={"ID":"827a5779-ed1a-4f71-b4da-9231c9f7962c","Type":"ContainerDied","Data":"64dcbb87f77fde6d6289c9d0ac24102ecf2140799e59e8ad18c9ad5eec8f4ece"}
Jul 17 19:19:16.208087 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.208068    2563 scope.go:117] "RemoveContainer" containerID="b82635e78501e1358afe68b58c2096d7f507ea05a8a5bab5f138220ee3bf65ee"
Jul 17 19:19:16.215768 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.215750    2563 scope.go:117] "RemoveContainer" containerID="5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981"
Jul 17 19:19:16.223519 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.223498    2563 scope.go:117] "RemoveContainer" containerID="221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54"
Jul 17 19:19:16.228406 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.228383    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"]
Jul 17 19:19:16.231373 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.231353    2563 scope.go:117] "RemoveContainer" containerID="5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981"
Jul 17 19:19:16.231641 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:16.231624    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981\": container with ID starting with 5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981 not found: ID does not exist" containerID="5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981"
Jul 17 19:19:16.231685 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.231650    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981"} err="failed to get container status \"5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981\": rpc error: code = NotFound desc = could not find container \"5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981\": container with ID starting with 5419a9cf7960e762d78a1feaf5d3e05683eb3664544f3b05d2cfee3792101981 not found: ID does not exist"
Jul 17 19:19:16.231685 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.231668    2563 scope.go:117] "RemoveContainer" containerID="221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54"
Jul 17 19:19:16.231922 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:16.231906    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54\": container with ID starting with 221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54 not found: ID does not exist" containerID="221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54"
Jul 17 19:19:16.231978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.231924    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54"} err="failed to get container status \"221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54\": rpc error: code = NotFound desc = could not find container \"221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54\": container with ID starting with 221e405a2b1ee04ddbc50ea98db4fb1188888dc6b6bb5c0e10f33b3f7b29ca54 not found: ID does not exist"
Jul 17 19:19:16.234429 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.234405    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bd0f3bb1-openshift-default-584dfdcf59-5kpvh"]
Jul 17 19:19:16.246831 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.246807    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"]
Jul 17 19:19:16.251408 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:16.251388    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-wmq29"]
Jul 17 19:19:17.097963 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:17.097929    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:17.206820 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:17.206786    2563 scope.go:117] "RemoveContainer" containerID="8a99446849f5b4fc0741e79b4ebb5cdca76dce0225a22dd6ea26e859f5d49bdf"
Jul 17 19:19:17.207283 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:17.206980    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2_odh-ai-gateway-infra(2613d6de-4e70-400e-b27f-13c3396b996a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" podUID="2613d6de-4e70-400e-b27f-13c3396b996a"
Jul 17 19:19:17.694590 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:17.694554    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" path="/var/lib/kubelet/pods/827a5779-ed1a-4f71-b4da-9231c9f7962c/volumes"
Jul 17 19:19:17.694907 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:17.694893    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" path="/var/lib/kubelet/pods/de884e27-f8a8-4bfc-89cd-7f587e2189f7/volumes"
Jul 17 19:19:18.122483 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:18.122444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:18.242403 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:18.242366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:18.373401 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:18.373316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:18.509428 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:18.509397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:24.724248 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:24.724209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:24.801694 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:24.801661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:25.178824 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:25.178784    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:27.105745 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:27.105706    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:31.519092 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.519008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:31.691106 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.691073    2563 scope.go:117] "RemoveContainer" containerID="8a99446849f5b4fc0741e79b4ebb5cdca76dce0225a22dd6ea26e859f5d49bdf"
Jul 17 19:19:31.810201 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810104    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"]
Jul 17 19:19:31.810438 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810425    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy"
Jul 17 19:19:31.810486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810441    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy"
Jul 17 19:19:31.810486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810459    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy"
Jul 17 19:19:31.810486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810466    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy"
Jul 17 19:19:31.810486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810475    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:31.810486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810483    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:31.810640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810496    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:31.810640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810502    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:31.810640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810549    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:31.810640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810559    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy"
Jul 17 19:19:31.810640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810566    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:31.810640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.810572    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="827a5779-ed1a-4f71-b4da-9231c9f7962c" containerName="istio-proxy"
Jul 17 19:19:31.814861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.814839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.817750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.817732    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-010d52ef-openshift-default-dockercfg-pj55x\""
Jul 17 19:19:31.825913 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.825890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"]
Jul 17 19:19:31.840246 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.840212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:31.955431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-token\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.955613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786c1c60-8786-40d0-9cce-69a15d8de50d-istiod-ca-cert\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.955613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x45b6\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-kube-api-access-x45b6\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.955613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-certs\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.955613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-podinfo\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.955613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-socket\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.955613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-data\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.955863 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-envoy\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.955863 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.955685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-credential-socket\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:31.965592 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:31.965559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:32.056339 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-data\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-envoy\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-credential-socket\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-token\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786c1c60-8786-40d0-9cce-69a15d8de50d-istiod-ca-cert\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x45b6\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-kube-api-access-x45b6\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-certs\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-podinfo\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056904 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-socket\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.056904 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-data\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.057009 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.056983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-credential-socket\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.057064 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.057036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-certs\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.057147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.057094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-socket\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.057241 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.057218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786c1c60-8786-40d0-9cce-69a15d8de50d-istiod-ca-cert\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.058766 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.058747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-envoy\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.059306 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.059288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-podinfo\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.068980 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.068918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-token\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.068980 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.068953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x45b6\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-kube-api-access-x45b6\") pod \"e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") " pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.124579 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.124544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:32.270266 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.270229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" event={"ID":"2613d6de-4e70-400e-b27f-13c3396b996a","Type":"ContainerStarted","Data":"9b0491be8eeaccec8900afb7a39232dade7bda67a3ff9ca3c2ef3a7d1689b089"}
Jul 17 19:19:32.271616 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.271596    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"]
Jul 17 19:19:32.273249 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:19:32.273224    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786c1c60_8786_40d0_9cce_69a15d8de50d.slice/crio-a1424ea37858baddb271d088b428d1b7f61a1ca3081c842f8c523fc90e790e69 WatchSource:0}: Error finding container a1424ea37858baddb271d088b428d1b7f61a1ca3081c842f8c523fc90e790e69: Status 404 returned error can't find the container with id a1424ea37858baddb271d088b428d1b7f61a1ca3081c842f8c523fc90e790e69
Jul 17 19:19:32.275202 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.275169    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:19:32.275295 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.275238    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:19:32.275295 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.275266    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:19:32.306240 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:32.306198    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" podStartSLOduration=19.306184125 podStartE2EDuration="19.306184125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:19:32.303844413 +0000 UTC m=+2781.214671173" watchObservedRunningTime="2026-07-17 19:19:32.306184125 +0000 UTC m=+2781.217010863"
Jul 17 19:19:33.276597 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:33.276562    2563 generic.go:356] "Generic (PLEG): container finished" podID="2613d6de-4e70-400e-b27f-13c3396b996a" containerID="9b0491be8eeaccec8900afb7a39232dade7bda67a3ff9ca3c2ef3a7d1689b089" exitCode=6
Jul 17 19:19:33.277028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:33.276642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" event={"ID":"2613d6de-4e70-400e-b27f-13c3396b996a","Type":"ContainerDied","Data":"9b0491be8eeaccec8900afb7a39232dade7bda67a3ff9ca3c2ef3a7d1689b089"}
Jul 17 19:19:33.277028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:33.276746    2563 scope.go:117] "RemoveContainer" containerID="8a99446849f5b4fc0741e79b4ebb5cdca76dce0225a22dd6ea26e859f5d49bdf"
Jul 17 19:19:33.277182 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:33.277057    2563 scope.go:117] "RemoveContainer" containerID="9b0491be8eeaccec8900afb7a39232dade7bda67a3ff9ca3c2ef3a7d1689b089"
Jul 17 19:19:33.277366 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:33.277333    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2_odh-ai-gateway-infra(2613d6de-4e70-400e-b27f-13c3396b996a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" podUID="2613d6de-4e70-400e-b27f-13c3396b996a"
Jul 17 19:19:33.278267 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:33.278246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" event={"ID":"786c1c60-8786-40d0-9cce-69a15d8de50d","Type":"ContainerStarted","Data":"c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0"}
Jul 17 19:19:33.278340 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:33.278291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" event={"ID":"786c1c60-8786-40d0-9cce-69a15d8de50d","Type":"ContainerStarted","Data":"a1424ea37858baddb271d088b428d1b7f61a1ca3081c842f8c523fc90e790e69"}
Jul 17 19:19:33.315855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:33.315821    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"]
Jul 17 19:19:33.318794 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:33.318743    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" podStartSLOduration=2.318728428 podStartE2EDuration="2.318728428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:19:33.317477039 +0000 UTC m=+2782.228303800" watchObservedRunningTime="2026-07-17 19:19:33.318728428 +0000 UTC m=+2782.229555225"
Jul 17 19:19:34.125208 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:34.125168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:34.130233 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:34.130208    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:34.284382 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:34.283895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:34.285137 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:34.285104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:34.436942 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:34.436917    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"
Jul 17 19:19:34.580231 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:34.580192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2613d6de-4e70-400e-b27f-13c3396b996a-maas-api-service-ca\") pod \"2613d6de-4e70-400e-b27f-13c3396b996a\" (UID: \"2613d6de-4e70-400e-b27f-13c3396b996a\") "
Jul 17 19:19:34.580564 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:34.580541    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2613d6de-4e70-400e-b27f-13c3396b996a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2613d6de-4e70-400e-b27f-13c3396b996a" (UID: "2613d6de-4e70-400e-b27f-13c3396b996a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:19:34.681694 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:34.681606    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2613d6de-4e70-400e-b27f-13c3396b996a-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:19:35.289003 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.288970    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"
Jul 17 19:19:35.289003 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.288987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2" event={"ID":"2613d6de-4e70-400e-b27f-13c3396b996a","Type":"ContainerDied","Data":"27ea0d4ea9885d6735165d31da7c0a59e8bb70f3de4a459026f5558e41fc3a1a"}
Jul 17 19:19:35.289617 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.289059    2563 scope.go:117] "RemoveContainer" containerID="9b0491be8eeaccec8900afb7a39232dade7bda67a3ff9ca3c2ef3a7d1689b089"
Jul 17 19:19:35.324013 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.323979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:35.339225 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.339196    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"]
Jul 17 19:19:35.349019 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.348996    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7zsq2"]
Jul 17 19:19:35.430053 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430015    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"]
Jul 17 19:19:35.430319 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430307    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:19:35.430364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430321    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:19:35.430364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430330    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:35.430364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430336    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:35.430364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430350    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:19:35.430364 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430355    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:19:35.430517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430399    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:19:35.430517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430408    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:19:35.430517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430414    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:19:35.430517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.430422    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="de884e27-f8a8-4bfc-89cd-7f587e2189f7" containerName="revoke-keys"
Jul 17 19:19:35.434229 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.434212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.438062 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.438037    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:19:35.438440 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.438423    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:19:35.438709 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.438694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:19:35.438865 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.438849    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-010d52ef-serving-cert\""
Jul 17 19:19:35.445408 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.445386    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"]
Jul 17 19:19:35.446391 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.446351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:35.490136 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.490078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.490328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.490180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e00d9-c65d-4b50-a9e1-56941660cda9-openshift-service-ca\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.490328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.490236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjmh7\" (UniqueName: \"kubernetes.io/projected/0a2e00d9-c65d-4b50-a9e1-56941660cda9-kube-api-access-rjmh7\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.565091 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.565005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:35.591344 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.591308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e00d9-c65d-4b50-a9e1-56941660cda9-openshift-service-ca\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.591487 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.591366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjmh7\" (UniqueName: \"kubernetes.io/projected/0a2e00d9-c65d-4b50-a9e1-56941660cda9-kube-api-access-rjmh7\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.591487 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.591409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.591562 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:35.591520    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-serving-cert: secret "maas-api-e2e-mt-010d52ef-serving-cert" not found
Jul 17 19:19:35.591607 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:35.591597    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls podName:0a2e00d9-c65d-4b50-a9e1-56941660cda9 nodeName:}" failed. No retries permitted until 2026-07-17 19:19:36.091575676 +0000 UTC m=+2785.002402415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls") pod "maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h" (UID: "0a2e00d9-c65d-4b50-a9e1-56941660cda9") : secret "maas-api-e2e-mt-010d52ef-serving-cert" not found
Jul 17 19:19:35.592037 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.592016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e00d9-c65d-4b50-a9e1-56941660cda9-openshift-service-ca\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.602589 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.602562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjmh7\" (UniqueName: \"kubernetes.io/projected/0a2e00d9-c65d-4b50-a9e1-56941660cda9-kube-api-access-rjmh7\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:35.694301 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:35.694260    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" path="/var/lib/kubelet/pods/2613d6de-4e70-400e-b27f-13c3396b996a/volumes"
Jul 17 19:19:36.074110 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:36.074071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:36.095968 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:36.095937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:36.098606 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:36.098563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls\") pod \"maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:36.172964 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:36.172931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:36.345577 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:36.345546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:36.493378 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:36.493330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"]
Jul 17 19:19:36.495858 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:19:36.495833    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2e00d9_c65d_4b50_a9e1_56941660cda9.slice/crio-5e6c3a61e3e06d7e58579f4fee0e29d30b1144046045fd5c1d574dbff35ff5be WatchSource:0}: Error finding container 5e6c3a61e3e06d7e58579f4fee0e29d30b1144046045fd5c1d574dbff35ff5be: Status 404 returned error can't find the container with id 5e6c3a61e3e06d7e58579f4fee0e29d30b1144046045fd5c1d574dbff35ff5be
Jul 17 19:19:37.114374 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:37.114344    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:37.300543 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:37.300449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h" event={"ID":"0a2e00d9-c65d-4b50-a9e1-56941660cda9","Type":"ContainerStarted","Data":"580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef"}
Jul 17 19:19:37.300543 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:37.300521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h" event={"ID":"0a2e00d9-c65d-4b50-a9e1-56941660cda9","Type":"ContainerStarted","Data":"5e6c3a61e3e06d7e58579f4fee0e29d30b1144046045fd5c1d574dbff35ff5be"}
Jul 17 19:19:37.300754 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:37.300549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:39.838402 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:39.838365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:40.194500 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:40.194468    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:40.662723 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:40.662647    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h" podStartSLOduration=5.159947936 podStartE2EDuration="5.662622558s" totalImagesPullingTime="502.674622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:19:37.337493213 +0000 UTC m=+2786.248319992" watchObservedRunningTime="2026-07-17 19:19:40.662622558 +0000 UTC m=+2789.573449320"
Jul 17 19:19:40.662997 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:40.662948    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:19:43.311089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:43.311061    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:19:47.149236 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:47.149199    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:49.285275 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:49.285238    2563 patch_prober.go:28] interesting pod/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 17 19:19:49.285675 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:49.285301    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 17 19:19:49.353852 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:49.353818    2563 generic.go:356] "Generic (PLEG): container finished" podID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerID="c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0" exitCode=0
Jul 17 19:19:49.354026 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:49.353892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" event={"ID":"786c1c60-8786-40d0-9cce-69a15d8de50d","Type":"ContainerDied","Data":"c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0"}
Jul 17 19:19:49.354311 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:49.354297    2563 scope.go:117] "RemoveContainer" containerID="c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0"
Jul 17 19:19:49.355321 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:49.355292    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:19:49.355420 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:49.355401    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:19:49.355458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:49.355444    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:19:50.360190 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:50.360153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" event={"ID":"786c1c60-8786-40d0-9cce-69a15d8de50d","Type":"ContainerStarted","Data":"1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3"}
Jul 17 19:19:51.125315 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:51.125269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:51.125568 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:51.125543    2563 patch_prober.go:28] interesting pod/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 17 19:19:51.125660 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:51.125601    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 17 19:19:52.125222 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:52.125068    2563 patch_prober.go:28] interesting pod/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 17 19:19:52.125222 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:52.125165    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 17 19:19:53.125552 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:53.125504    2563 patch_prober.go:28] interesting pod/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 17 19:19:53.126027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:53.125578    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 17 19:19:54.125014 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:54.124972    2563 patch_prober.go:28] interesting pod/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 17 19:19:54.125230 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:54.125031    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 17 19:19:54.168690 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:54.168652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:19:55.129709 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:55.129675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:55.129947 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:55.129925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:55.130756 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:55.130741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:19:55.182949 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:55.182917    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:57.156682 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:19:57.156651    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:19:57.418490 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:19:57.418398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:20:07.191647 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:07.191610    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:20:10.160960 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:10.160911    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde884e27_f8a8_4bfc_89cd_7f587e2189f7.slice/crio-1b06cae7e187beaa4db57978d18d1cbbb2f08109c784fc3bda099a99790dc18d\": RecentStats: unable to find data in memory cache]"
Jul 17 19:20:13.993092 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:13.993056    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"]
Jul 17 19:20:13.993545 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:13.993366    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:20:13.993545 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:13.993379    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2613d6de-4e70-400e-b27f-13c3396b996a" containerName="revoke-keys"
Jul 17 19:20:13.998113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:13.998089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"
Jul 17 19:20:14.005147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:14.005103    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:20:14.011656 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:14.011628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"]
Jul 17 19:20:14.029747 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:14.029697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a86e372-0a5d-455d-8858-39ec3a99aaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g\" (UID: \"4a86e372-0a5d-455d-8858-39ec3a99aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"
Jul 17 19:20:14.131072 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:14.131029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a86e372-0a5d-455d-8858-39ec3a99aaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g\" (UID: \"4a86e372-0a5d-455d-8858-39ec3a99aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"
Jul 17 19:20:14.131688 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:14.131667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a86e372-0a5d-455d-8858-39ec3a99aaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g\" (UID: \"4a86e372-0a5d-455d-8858-39ec3a99aaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"
Jul 17 19:20:14.308812 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:14.308729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"
Jul 17 19:20:14.447168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:14.447135    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"]
Jul 17 19:20:14.449730 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:20:14.449696    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a86e372_0a5d_455d_8858_39ec3a99aaa2.slice/crio-079a5cc0c945db7711657408f41e766029ae224c624a49f74ecae6f4924c2f6e WatchSource:0}: Error finding container 079a5cc0c945db7711657408f41e766029ae224c624a49f74ecae6f4924c2f6e: Status 404 returned error can't find the container with id 079a5cc0c945db7711657408f41e766029ae224c624a49f74ecae6f4924c2f6e
Jul 17 19:20:14.465493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:14.465468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g" event={"ID":"4a86e372-0a5d-455d-8858-39ec3a99aaa2","Type":"ContainerStarted","Data":"079a5cc0c945db7711657408f41e766029ae224c624a49f74ecae6f4924c2f6e"}
Jul 17 19:20:15.143173 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:15.143137    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:20:15.471804 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:15.471762    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerID="198851bf88a503822265a7cbbb10937399c41c96011db7080e6d7e3fca9c4362" exitCode=6
Jul 17 19:20:15.471995 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:15.471848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g" event={"ID":"4a86e372-0a5d-455d-8858-39ec3a99aaa2","Type":"ContainerDied","Data":"198851bf88a503822265a7cbbb10937399c41c96011db7080e6d7e3fca9c4362"}
Jul 17 19:20:15.472046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:15.472026    2563 scope.go:117] "RemoveContainer" containerID="198851bf88a503822265a7cbbb10937399c41c96011db7080e6d7e3fca9c4362"
Jul 17 19:20:16.477424 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:16.477386    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerID="b75d137651166c8008c4b2abf9b6cea380f09e2dfc3d9cb84bed539cb6fa9d35" exitCode=6
Jul 17 19:20:16.477837 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:16.477477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g" event={"ID":"4a86e372-0a5d-455d-8858-39ec3a99aaa2","Type":"ContainerDied","Data":"b75d137651166c8008c4b2abf9b6cea380f09e2dfc3d9cb84bed539cb6fa9d35"}
Jul 17 19:20:16.477837 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:16.477571    2563 scope.go:117] "RemoveContainer" containerID="198851bf88a503822265a7cbbb10937399c41c96011db7080e6d7e3fca9c4362"
Jul 17 19:20:16.477837 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:16.477722    2563 scope.go:117] "RemoveContainer" containerID="b75d137651166c8008c4b2abf9b6cea380f09e2dfc3d9cb84bed539cb6fa9d35"
Jul 17 19:20:16.477989 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:16.477958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g_odh-ai-gateway-infra(4a86e372-0a5d-455d-8858-39ec3a99aaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2"
Jul 17 19:20:17.483319 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:17.483290    2563 scope.go:117] "RemoveContainer" containerID="b75d137651166c8008c4b2abf9b6cea380f09e2dfc3d9cb84bed539cb6fa9d35"
Jul 17 19:20:17.483705 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:17.483471    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g_odh-ai-gateway-infra(4a86e372-0a5d-455d-8858-39ec3a99aaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2"
Jul 17 19:20:23.503070 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:23.503018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:20:23.604579 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:23.604545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:20:28.596976 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.596938    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"]
Jul 17 19:20:28.597603 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.597273    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h" podUID="0a2e00d9-c65d-4b50-a9e1-56941660cda9" containerName="maas-api" containerID="cri-o://580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef" gracePeriod=30
Jul 17 19:20:28.622162 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.622097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:20:28.847949 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.847891    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:20:28.957256 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.957220    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls\") pod \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") "
Jul 17 19:20:28.957256 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.957268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e00d9-c65d-4b50-a9e1-56941660cda9-openshift-service-ca\") pod \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") "
Jul 17 19:20:28.957494 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.957320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjmh7\" (UniqueName: \"kubernetes.io/projected/0a2e00d9-c65d-4b50-a9e1-56941660cda9-kube-api-access-rjmh7\") pod \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\" (UID: \"0a2e00d9-c65d-4b50-a9e1-56941660cda9\") "
Jul 17 19:20:28.957677 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.957651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2e00d9-c65d-4b50-a9e1-56941660cda9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a2e00d9-c65d-4b50-a9e1-56941660cda9" (UID: "0a2e00d9-c65d-4b50-a9e1-56941660cda9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:20:28.959553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.959533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a2e00d9-c65d-4b50-a9e1-56941660cda9" (UID: "0a2e00d9-c65d-4b50-a9e1-56941660cda9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:20:28.959553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:28.959535    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2e00d9-c65d-4b50-a9e1-56941660cda9-kube-api-access-rjmh7" (OuterVolumeSpecName: "kube-api-access-rjmh7") pod "0a2e00d9-c65d-4b50-a9e1-56941660cda9" (UID: "0a2e00d9-c65d-4b50-a9e1-56941660cda9"). InnerVolumeSpecName "kube-api-access-rjmh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:20:29.058331 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.058297    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjmh7\" (UniqueName: \"kubernetes.io/projected/0a2e00d9-c65d-4b50-a9e1-56941660cda9-kube-api-access-rjmh7\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:20:29.058331 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.058328    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a2e00d9-c65d-4b50-a9e1-56941660cda9-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:20:29.058331 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.058337    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2e00d9-c65d-4b50-a9e1-56941660cda9-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:20:29.531253 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.531214    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a2e00d9-c65d-4b50-a9e1-56941660cda9" containerID="580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef" exitCode=0
Jul 17 19:20:29.531429 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.531261    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"
Jul 17 19:20:29.531429 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.531303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h" event={"ID":"0a2e00d9-c65d-4b50-a9e1-56941660cda9","Type":"ContainerDied","Data":"580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef"}
Jul 17 19:20:29.531429 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.531364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h" event={"ID":"0a2e00d9-c65d-4b50-a9e1-56941660cda9","Type":"ContainerDied","Data":"5e6c3a61e3e06d7e58579f4fee0e29d30b1144046045fd5c1d574dbff35ff5be"}
Jul 17 19:20:29.531429 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.531386    2563 scope.go:117] "RemoveContainer" containerID="580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef"
Jul 17 19:20:29.540541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.540522    2563 scope.go:117] "RemoveContainer" containerID="580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef"
Jul 17 19:20:29.540811 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:29.540781    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef\": container with ID starting with 580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef not found: ID does not exist" containerID="580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef"
Jul 17 19:20:29.540865 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.540817    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef"} err="failed to get container status \"580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef\": rpc error: code = NotFound desc = could not find container \"580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef\": container with ID starting with 580aaacc25c37f634c99872afb2ab6462191741185485ed37325996b325602ef not found: ID does not exist"
Jul 17 19:20:29.551872 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.551845    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"]
Jul 17 19:20:29.555078 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.555055    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-010d52ef-bfd5f8db7-g548h"]
Jul 17 19:20:29.698501 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:29.695342    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2e00d9-c65d-4b50-a9e1-56941660cda9" path="/var/lib/kubelet/pods/0a2e00d9-c65d-4b50-a9e1-56941660cda9/volumes"
Jul 17 19:20:32.496900 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.496864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:20:32.689253 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.689221    2563 scope.go:117] "RemoveContainer" containerID="b75d137651166c8008c4b2abf9b6cea380f09e2dfc3d9cb84bed539cb6fa9d35"
Jul 17 19:20:32.854962 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.854926    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"]
Jul 17 19:20:32.855248 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.855234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2e00d9-c65d-4b50-a9e1-56941660cda9" containerName="maas-api"
Jul 17 19:20:32.855248 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.855248    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2e00d9-c65d-4b50-a9e1-56941660cda9" containerName="maas-api"
Jul 17 19:20:32.855350 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.855300    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2e00d9-c65d-4b50-a9e1-56941660cda9" containerName="maas-api"
Jul 17 19:20:32.859481 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.859463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"
Jul 17 19:20:32.868219 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.868190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"]
Jul 17 19:20:32.991450 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:32.991421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7e67e5-260f-4cb6-92d2-ccb58f79aa42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm\" (UID: \"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"
Jul 17 19:20:33.092170 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.092103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7e67e5-260f-4cb6-92d2-ccb58f79aa42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm\" (UID: \"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"
Jul 17 19:20:33.092821 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.092793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7e67e5-260f-4cb6-92d2-ccb58f79aa42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm\" (UID: \"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"
Jul 17 19:20:33.170046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.169955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"
Jul 17 19:20:33.273915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.273884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:20:33.307728 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.307674    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"]
Jul 17 19:20:33.310051 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:20:33.310021    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7e67e5_260f_4cb6_92d2_ccb58f79aa42.slice/crio-dde387691b3c230dcbf7433be64a64a1d83f5491d5f77df813e881e43ad2a48a WatchSource:0}: Error finding container dde387691b3c230dcbf7433be64a64a1d83f5491d5f77df813e881e43ad2a48a: Status 404 returned error can't find the container with id dde387691b3c230dcbf7433be64a64a1d83f5491d5f77df813e881e43ad2a48a
Jul 17 19:20:33.550080 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.550049    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerID="399e99bf3ae289593ffaf7ed226e2a15cc7615c2615c87a89fec6d018e45aeb5" exitCode=6
Jul 17 19:20:33.550478 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.550142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g" event={"ID":"4a86e372-0a5d-455d-8858-39ec3a99aaa2","Type":"ContainerDied","Data":"399e99bf3ae289593ffaf7ed226e2a15cc7615c2615c87a89fec6d018e45aeb5"}
Jul 17 19:20:33.550478 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.550210    2563 scope.go:117] "RemoveContainer" containerID="b75d137651166c8008c4b2abf9b6cea380f09e2dfc3d9cb84bed539cb6fa9d35"
Jul 17 19:20:33.550618 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.550517    2563 scope.go:117] "RemoveContainer" containerID="399e99bf3ae289593ffaf7ed226e2a15cc7615c2615c87a89fec6d018e45aeb5"
Jul 17 19:20:33.550776 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:33.550752    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g_odh-ai-gateway-infra(4a86e372-0a5d-455d-8858-39ec3a99aaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2"
Jul 17 19:20:33.551726 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.551701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" event={"ID":"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42","Type":"ContainerStarted","Data":"527b623f36e4fca2f99160bce967c2fe7120ac7172c698ed23d4205c9b2ced76"}
Jul 17 19:20:33.551839 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.551753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" event={"ID":"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42","Type":"ContainerStarted","Data":"dde387691b3c230dcbf7433be64a64a1d83f5491d5f77df813e881e43ad2a48a"}
Jul 17 19:20:33.581534 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:33.581465    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" podStartSLOduration=1.581449117 podStartE2EDuration="1.581449117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:20:33.580730477 +0000 UTC m=+2842.491557258" watchObservedRunningTime="2026-07-17 19:20:33.581449117 +0000 UTC m=+2842.492275859"
Jul 17 19:20:34.559267 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:34.559228    2563 generic.go:356] "Generic (PLEG): container finished" podID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerID="527b623f36e4fca2f99160bce967c2fe7120ac7172c698ed23d4205c9b2ced76" exitCode=6
Jul 17 19:20:34.559714 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:34.559309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" event={"ID":"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42","Type":"ContainerDied","Data":"527b623f36e4fca2f99160bce967c2fe7120ac7172c698ed23d4205c9b2ced76"}
Jul 17 19:20:34.559714 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:34.559490    2563 scope.go:117] "RemoveContainer" containerID="527b623f36e4fca2f99160bce967c2fe7120ac7172c698ed23d4205c9b2ced76"
Jul 17 19:20:34.575612 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:34.575576    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"]
Jul 17 19:20:34.710143 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:34.710104    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"
Jul 17 19:20:34.808986 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:34.808956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a86e372-0a5d-455d-8858-39ec3a99aaa2-maas-api-service-ca\") pod \"4a86e372-0a5d-455d-8858-39ec3a99aaa2\" (UID: \"4a86e372-0a5d-455d-8858-39ec3a99aaa2\") "
Jul 17 19:20:34.809389 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:34.809344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a86e372-0a5d-455d-8858-39ec3a99aaa2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a86e372-0a5d-455d-8858-39ec3a99aaa2" (UID: "4a86e372-0a5d-455d-8858-39ec3a99aaa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:20:34.910448 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:34.910419    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a86e372-0a5d-455d-8858-39ec3a99aaa2-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:20:35.564362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.564277    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"
Jul 17 19:20:35.564362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.564273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g" event={"ID":"4a86e372-0a5d-455d-8858-39ec3a99aaa2","Type":"ContainerDied","Data":"079a5cc0c945db7711657408f41e766029ae224c624a49f74ecae6f4924c2f6e"}
Jul 17 19:20:35.564862 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.564414    2563 scope.go:117] "RemoveContainer" containerID="399e99bf3ae289593ffaf7ed226e2a15cc7615c2615c87a89fec6d018e45aeb5"
Jul 17 19:20:35.566314 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.566289    2563 generic.go:356] "Generic (PLEG): container finished" podID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerID="ceb1b3e16766e124a30b8958d1482c7f946a92f97bfc0c8bf47850a6e6a6e6bd" exitCode=6
Jul 17 19:20:35.566395 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.566324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" event={"ID":"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42","Type":"ContainerDied","Data":"ceb1b3e16766e124a30b8958d1482c7f946a92f97bfc0c8bf47850a6e6a6e6bd"}
Jul 17 19:20:35.566613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.566572    2563 scope.go:117] "RemoveContainer" containerID="ceb1b3e16766e124a30b8958d1482c7f946a92f97bfc0c8bf47850a6e6a6e6bd"
Jul 17 19:20:35.566870 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:35.566847    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm_odh-ai-gateway-infra(3a7e67e5-260f-4cb6-92d2-ccb58f79aa42)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42"
Jul 17 19:20:35.573743 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.573629    2563 scope.go:117] "RemoveContainer" containerID="527b623f36e4fca2f99160bce967c2fe7120ac7172c698ed23d4205c9b2ced76"
Jul 17 19:20:35.600447 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.600418    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"]
Jul 17 19:20:35.605857 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.605836    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wnt9g"]
Jul 17 19:20:35.699952 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:35.699913    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" path="/var/lib/kubelet/pods/4a86e372-0a5d-455d-8858-39ec3a99aaa2/volumes"
Jul 17 19:20:36.573823 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:36.573785    2563 scope.go:117] "RemoveContainer" containerID="ceb1b3e16766e124a30b8958d1482c7f946a92f97bfc0c8bf47850a6e6a6e6bd"
Jul 17 19:20:36.574334 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:36.574049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm_odh-ai-gateway-infra(3a7e67e5-260f-4cb6-92d2-ccb58f79aa42)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42"
Jul 17 19:20:38.369397 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:38.369359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:20:40.707954 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:40.707913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:20:40.808256 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:40.808217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:20:45.822400 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.822358    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"]
Jul 17 19:20:45.822881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.822655    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:20:45.822881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.822666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:20:45.822881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.822675    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:20:45.822881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.822681    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:20:45.822881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.822738    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:20:45.822881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.822748    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:20:45.826999 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.826982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"
Jul 17 19:20:45.834941 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:45.834912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"]
Jul 17 19:20:46.000728 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:46.000683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1bacbf-3810-4aaa-92dd-841281deb47c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l\" (UID: \"af1bacbf-3810-4aaa-92dd-841281deb47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"
Jul 17 19:20:46.101598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:46.101562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1bacbf-3810-4aaa-92dd-841281deb47c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l\" (UID: \"af1bacbf-3810-4aaa-92dd-841281deb47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"
Jul 17 19:20:46.102206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:46.102187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1bacbf-3810-4aaa-92dd-841281deb47c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l\" (UID: \"af1bacbf-3810-4aaa-92dd-841281deb47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"
Jul 17 19:20:46.138700 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:46.138668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"
Jul 17 19:20:46.293934 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:46.293902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"]
Jul 17 19:20:46.297743 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:20:46.297715    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1bacbf_3810_4aaa_92dd_841281deb47c.slice/crio-b7f2699fd6c7702b1a6ef2b6c3d4b9774ca6ca54c45de5941a251db40727c531 WatchSource:0}: Error finding container b7f2699fd6c7702b1a6ef2b6c3d4b9774ca6ca54c45de5941a251db40727c531: Status 404 returned error can't find the container with id b7f2699fd6c7702b1a6ef2b6c3d4b9774ca6ca54c45de5941a251db40727c531
Jul 17 19:20:46.619612 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:46.619571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" event={"ID":"af1bacbf-3810-4aaa-92dd-841281deb47c","Type":"ContainerStarted","Data":"6062e0e0622857e3a87169724875948cb4544b3b2c8945d8a0f71ddc3ec354a0"}
Jul 17 19:20:46.619788 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:46.619630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" event={"ID":"af1bacbf-3810-4aaa-92dd-841281deb47c","Type":"ContainerStarted","Data":"b7f2699fd6c7702b1a6ef2b6c3d4b9774ca6ca54c45de5941a251db40727c531"}
Jul 17 19:20:46.635286 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:46.635238    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" podStartSLOduration=1.635221762 podStartE2EDuration="1.635221762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:20:46.63396815 +0000 UTC m=+2855.544794910" watchObservedRunningTime="2026-07-17 19:20:46.635221762 +0000 UTC m=+2855.546048523"
Jul 17 19:20:47.625685 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:47.625652    2563 generic.go:356] "Generic (PLEG): container finished" podID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerID="6062e0e0622857e3a87169724875948cb4544b3b2c8945d8a0f71ddc3ec354a0" exitCode=6
Jul 17 19:20:47.626181 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:47.625713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" event={"ID":"af1bacbf-3810-4aaa-92dd-841281deb47c","Type":"ContainerDied","Data":"6062e0e0622857e3a87169724875948cb4544b3b2c8945d8a0f71ddc3ec354a0"}
Jul 17 19:20:47.626181 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:47.625930    2563 scope.go:117] "RemoveContainer" containerID="6062e0e0622857e3a87169724875948cb4544b3b2c8945d8a0f71ddc3ec354a0"
Jul 17 19:20:48.631695 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:48.631654    2563 generic.go:356] "Generic (PLEG): container finished" podID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerID="07f7cf803ea54fb107af6221aa327d2e2531f3980371f473dd65936b7a178cdc" exitCode=6
Jul 17 19:20:48.632147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:48.631740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" event={"ID":"af1bacbf-3810-4aaa-92dd-841281deb47c","Type":"ContainerDied","Data":"07f7cf803ea54fb107af6221aa327d2e2531f3980371f473dd65936b7a178cdc"}
Jul 17 19:20:48.632147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:48.631804    2563 scope.go:117] "RemoveContainer" containerID="6062e0e0622857e3a87169724875948cb4544b3b2c8945d8a0f71ddc3ec354a0"
Jul 17 19:20:48.632147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:48.632043    2563 scope.go:117] "RemoveContainer" containerID="07f7cf803ea54fb107af6221aa327d2e2531f3980371f473dd65936b7a178cdc"
Jul 17 19:20:48.632321 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:48.632299    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l_odh-ai-gateway-infra(af1bacbf-3810-4aaa-92dd-841281deb47c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c"
Jul 17 19:20:49.637549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:49.637520    2563 scope.go:117] "RemoveContainer" containerID="07f7cf803ea54fb107af6221aa327d2e2531f3980371f473dd65936b7a178cdc"
Jul 17 19:20:49.637937 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:49.637704    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l_odh-ai-gateway-infra(af1bacbf-3810-4aaa-92dd-841281deb47c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c"
Jul 17 19:20:50.689677 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:50.689640    2563 scope.go:117] "RemoveContainer" containerID="ceb1b3e16766e124a30b8958d1482c7f946a92f97bfc0c8bf47850a6e6a6e6bd"
Jul 17 19:20:51.647127 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:51.647095    2563 generic.go:356] "Generic (PLEG): container finished" podID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerID="6546161851c0afe40f552dd178bd6b85bddcf9c6935732bf72e8e6cee2014662" exitCode=6
Jul 17 19:20:51.647331 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:51.647174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" event={"ID":"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42","Type":"ContainerDied","Data":"6546161851c0afe40f552dd178bd6b85bddcf9c6935732bf72e8e6cee2014662"}
Jul 17 19:20:51.647331 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:51.647238    2563 scope.go:117] "RemoveContainer" containerID="ceb1b3e16766e124a30b8958d1482c7f946a92f97bfc0c8bf47850a6e6a6e6bd"
Jul 17 19:20:51.647557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:51.647541    2563 scope.go:117] "RemoveContainer" containerID="6546161851c0afe40f552dd178bd6b85bddcf9c6935732bf72e8e6cee2014662"
Jul 17 19:20:51.647786 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:20:51.647759    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm_odh-ai-gateway-infra(3a7e67e5-260f-4cb6-92d2-ccb58f79aa42)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42"
Jul 17 19:20:52.672520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:52.672472    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"]
Jul 17 19:20:52.808705 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:52.808683    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"
Jul 17 19:20:52.960199 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:52.960082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7e67e5-260f-4cb6-92d2-ccb58f79aa42-maas-api-service-ca\") pod \"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42\" (UID: \"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42\") "
Jul 17 19:20:52.960450 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:52.960426    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7e67e5-260f-4cb6-92d2-ccb58f79aa42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" (UID: "3a7e67e5-260f-4cb6-92d2-ccb58f79aa42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:20:53.061061 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:53.061024    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7e67e5-260f-4cb6-92d2-ccb58f79aa42-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:20:53.658577 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:53.658536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm" event={"ID":"3a7e67e5-260f-4cb6-92d2-ccb58f79aa42","Type":"ContainerDied","Data":"dde387691b3c230dcbf7433be64a64a1d83f5491d5f77df813e881e43ad2a48a"}
Jul 17 19:20:53.658755 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:53.658638    2563 scope.go:117] "RemoveContainer" containerID="6546161851c0afe40f552dd178bd6b85bddcf9c6935732bf72e8e6cee2014662"
Jul 17 19:20:53.658755 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:53.658553    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"
Jul 17 19:20:53.684598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:53.684568    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"]
Jul 17 19:20:53.688224 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:53.688199    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k5wqm"]
Jul 17 19:20:53.694492 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:20:53.694462    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" path="/var/lib/kubelet/pods/3a7e67e5-260f-4cb6-92d2-ccb58f79aa42/volumes"
Jul 17 19:21:01.155946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.155914    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"]
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156220    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156233    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156242    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156247    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156266    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156272    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156326    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a86e372-0a5d-455d-8858-39ec3a99aaa2" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156333    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156342    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:01.156409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.156347    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:01.160545 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.160526    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"
Jul 17 19:21:01.166907 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.166881    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"]
Jul 17 19:21:01.225831 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.225796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320724fb-bbeb-44a0-8110-a3aaccc6348d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj\" (UID: \"320724fb-bbeb-44a0-8110-a3aaccc6348d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"
Jul 17 19:21:01.326919 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.326874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320724fb-bbeb-44a0-8110-a3aaccc6348d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj\" (UID: \"320724fb-bbeb-44a0-8110-a3aaccc6348d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"
Jul 17 19:21:01.327541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.327516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320724fb-bbeb-44a0-8110-a3aaccc6348d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj\" (UID: \"320724fb-bbeb-44a0-8110-a3aaccc6348d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"
Jul 17 19:21:01.471993 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.471863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"
Jul 17 19:21:01.608530 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.608361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"]
Jul 17 19:21:01.611196 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:21:01.611169    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320724fb_bbeb_44a0_8110_a3aaccc6348d.slice/crio-8239c5ba69b80760a74a2a6118a36ac900afa2a31b4db5c80cac5b7966dc9c4b WatchSource:0}: Error finding container 8239c5ba69b80760a74a2a6118a36ac900afa2a31b4db5c80cac5b7966dc9c4b: Status 404 returned error can't find the container with id 8239c5ba69b80760a74a2a6118a36ac900afa2a31b4db5c80cac5b7966dc9c4b
Jul 17 19:21:01.693499 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:01.693460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj" event={"ID":"320724fb-bbeb-44a0-8110-a3aaccc6348d","Type":"ContainerStarted","Data":"8239c5ba69b80760a74a2a6118a36ac900afa2a31b4db5c80cac5b7966dc9c4b"}
Jul 17 19:21:02.698148 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:02.698097    2563 generic.go:356] "Generic (PLEG): container finished" podID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerID="e81018a15c639779dd8ffc2adebfe4d904006388f269ed3061637137b5343b08" exitCode=6
Jul 17 19:21:02.698539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:02.698190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj" event={"ID":"320724fb-bbeb-44a0-8110-a3aaccc6348d","Type":"ContainerDied","Data":"e81018a15c639779dd8ffc2adebfe4d904006388f269ed3061637137b5343b08"}
Jul 17 19:21:02.698539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:02.698404    2563 scope.go:117] "RemoveContainer" containerID="e81018a15c639779dd8ffc2adebfe4d904006388f269ed3061637137b5343b08"
Jul 17 19:21:03.689535 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:03.689505    2563 scope.go:117] "RemoveContainer" containerID="07f7cf803ea54fb107af6221aa327d2e2531f3980371f473dd65936b7a178cdc"
Jul 17 19:21:03.704043 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:03.704015    2563 generic.go:356] "Generic (PLEG): container finished" podID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerID="e76d862de6089ab38b703d9aeed06f8554a3b806849601e785de380f5513dee2" exitCode=6
Jul 17 19:21:03.704408 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:03.704086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj" event={"ID":"320724fb-bbeb-44a0-8110-a3aaccc6348d","Type":"ContainerDied","Data":"e76d862de6089ab38b703d9aeed06f8554a3b806849601e785de380f5513dee2"}
Jul 17 19:21:03.704408 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:03.704157    2563 scope.go:117] "RemoveContainer" containerID="e81018a15c639779dd8ffc2adebfe4d904006388f269ed3061637137b5343b08"
Jul 17 19:21:03.704408 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:03.704335    2563 scope.go:117] "RemoveContainer" containerID="e76d862de6089ab38b703d9aeed06f8554a3b806849601e785de380f5513dee2"
Jul 17 19:21:03.704586 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:03.704562    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj_odh-ai-gateway-infra(320724fb-bbeb-44a0-8110-a3aaccc6348d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d"
Jul 17 19:21:04.709741 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:04.709704    2563 generic.go:356] "Generic (PLEG): container finished" podID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerID="1d3b1a9c47ed9d5c388d2e55332a0bd1481c1eb0fab872e246b43334fcc938f0" exitCode=6
Jul 17 19:21:04.710201 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:04.709780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" event={"ID":"af1bacbf-3810-4aaa-92dd-841281deb47c","Type":"ContainerDied","Data":"1d3b1a9c47ed9d5c388d2e55332a0bd1481c1eb0fab872e246b43334fcc938f0"}
Jul 17 19:21:04.710201 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:04.709842    2563 scope.go:117] "RemoveContainer" containerID="07f7cf803ea54fb107af6221aa327d2e2531f3980371f473dd65936b7a178cdc"
Jul 17 19:21:04.710201 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:04.710166    2563 scope.go:117] "RemoveContainer" containerID="1d3b1a9c47ed9d5c388d2e55332a0bd1481c1eb0fab872e246b43334fcc938f0"
Jul 17 19:21:04.710412 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:04.710388    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l_odh-ai-gateway-infra(af1bacbf-3810-4aaa-92dd-841281deb47c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c"
Jul 17 19:21:04.714102 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:04.714086    2563 scope.go:117] "RemoveContainer" containerID="e76d862de6089ab38b703d9aeed06f8554a3b806849601e785de380f5513dee2"
Jul 17 19:21:04.714313 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:04.714295    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj_odh-ai-gateway-infra(320724fb-bbeb-44a0-8110-a3aaccc6348d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d"
Jul 17 19:21:05.735513 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:05.735475    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"]
Jul 17 19:21:05.868937 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:05.868911    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"
Jul 17 19:21:05.958249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:05.958210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1bacbf-3810-4aaa-92dd-841281deb47c-maas-api-service-ca\") pod \"af1bacbf-3810-4aaa-92dd-841281deb47c\" (UID: \"af1bacbf-3810-4aaa-92dd-841281deb47c\") "
Jul 17 19:21:05.958550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:05.958527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1bacbf-3810-4aaa-92dd-841281deb47c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af1bacbf-3810-4aaa-92dd-841281deb47c" (UID: "af1bacbf-3810-4aaa-92dd-841281deb47c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:21:06.059020 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:06.058925    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1bacbf-3810-4aaa-92dd-841281deb47c-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:06.724362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:06.724332    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"
Jul 17 19:21:06.724563 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:06.724333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l" event={"ID":"af1bacbf-3810-4aaa-92dd-841281deb47c","Type":"ContainerDied","Data":"b7f2699fd6c7702b1a6ef2b6c3d4b9774ca6ca54c45de5941a251db40727c531"}
Jul 17 19:21:06.724563 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:06.724459    2563 scope.go:117] "RemoveContainer" containerID="1d3b1a9c47ed9d5c388d2e55332a0bd1481c1eb0fab872e246b43334fcc938f0"
Jul 17 19:21:06.748826 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:06.748797    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"]
Jul 17 19:21:06.754335 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:06.754311    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-cjv4l"]
Jul 17 19:21:07.693611 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:07.693578    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" path="/var/lib/kubelet/pods/af1bacbf-3810-4aaa-92dd-841281deb47c/volumes"
Jul 17 19:21:15.038855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:15.038813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:15.689352 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:15.689323    2563 scope.go:117] "RemoveContainer" containerID="e76d862de6089ab38b703d9aeed06f8554a3b806849601e785de380f5513dee2"
Jul 17 19:21:15.766042 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:15.766000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:16.767549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:16.767507    2563 generic.go:356] "Generic (PLEG): container finished" podID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerID="bd8439c53b809e59ac11fdaeb61e0cfbaf32728cecd062a1938f6330be9cad24" exitCode=6
Jul 17 19:21:16.768069 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:16.767567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj" event={"ID":"320724fb-bbeb-44a0-8110-a3aaccc6348d","Type":"ContainerDied","Data":"bd8439c53b809e59ac11fdaeb61e0cfbaf32728cecd062a1938f6330be9cad24"}
Jul 17 19:21:16.768069 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:16.767636    2563 scope.go:117] "RemoveContainer" containerID="e76d862de6089ab38b703d9aeed06f8554a3b806849601e785de380f5513dee2"
Jul 17 19:21:16.768069 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:16.767962    2563 scope.go:117] "RemoveContainer" containerID="bd8439c53b809e59ac11fdaeb61e0cfbaf32728cecd062a1938f6330be9cad24"
Jul 17 19:21:16.768258 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:16.768189    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj_odh-ai-gateway-infra(320724fb-bbeb-44a0-8110-a3aaccc6348d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d"
Jul 17 19:21:17.792707 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:17.792661    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"]
Jul 17 19:21:17.923793 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:17.923766    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"
Jul 17 19:21:17.952870 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:17.952836    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320724fb-bbeb-44a0-8110-a3aaccc6348d-maas-api-service-ca\") pod \"320724fb-bbeb-44a0-8110-a3aaccc6348d\" (UID: \"320724fb-bbeb-44a0-8110-a3aaccc6348d\") "
Jul 17 19:21:17.953280 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:17.953253    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320724fb-bbeb-44a0-8110-a3aaccc6348d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "320724fb-bbeb-44a0-8110-a3aaccc6348d" (UID: "320724fb-bbeb-44a0-8110-a3aaccc6348d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:21:18.054500 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:18.054409    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/320724fb-bbeb-44a0-8110-a3aaccc6348d-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:18.780273 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:18.780244    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"
Jul 17 19:21:18.780492 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:18.780246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj" event={"ID":"320724fb-bbeb-44a0-8110-a3aaccc6348d","Type":"ContainerDied","Data":"8239c5ba69b80760a74a2a6118a36ac900afa2a31b4db5c80cac5b7966dc9c4b"}
Jul 17 19:21:18.780492 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:18.780379    2563 scope.go:117] "RemoveContainer" containerID="bd8439c53b809e59ac11fdaeb61e0cfbaf32728cecd062a1938f6330be9cad24"
Jul 17 19:21:18.800912 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:18.800881    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"]
Jul 17 19:21:18.804736 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:18.804711    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g7jvj"]
Jul 17 19:21:19.694035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:19.693666    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" path="/var/lib/kubelet/pods/320724fb-bbeb-44a0-8110-a3aaccc6348d/volumes"
Jul 17 19:21:21.640673 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:21.640635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:34.239956 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:34.239914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:36.776501 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:36.776447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:36.833485 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:36.833450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:45.471216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.471180    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"]
Jul 17 19:21:45.471606 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.471438    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy" containerID="cri-o://1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3" gracePeriod=30
Jul 17 19:21:45.491622 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.491585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:45.705317 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.705280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:45.866592 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"]
Jul 17 19:21:45.866848 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866836    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:45.866901 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866850    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:45.866901 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866860    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:45.866901 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866866    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:45.866901 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866882    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:45.866901 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866887    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7e67e5-260f-4cb6-92d2-ccb58f79aa42" containerName="revoke-keys"
Jul 17 19:21:45.866901 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866900    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866906    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866911    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866917    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866929    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866974    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866981    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866986    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:45.867108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.866992    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:45.871074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.871056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"
Jul 17 19:21:45.873363 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.873339    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:21:45.873488 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.873452    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:21:45.879968 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.879940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"]
Jul 17 19:21:45.988539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:45.988500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4827f3d-1f61-4e76-ac0f-52a369bbc86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp\" (UID: \"c4827f3d-1f61-4e76-ac0f-52a369bbc86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"
Jul 17 19:21:46.090026 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.089982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4827f3d-1f61-4e76-ac0f-52a369bbc86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp\" (UID: \"c4827f3d-1f61-4e76-ac0f-52a369bbc86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"
Jul 17 19:21:46.090645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.090620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4827f3d-1f61-4e76-ac0f-52a369bbc86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp\" (UID: \"c4827f3d-1f61-4e76-ac0f-52a369bbc86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"
Jul 17 19:21:46.182233 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.182147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"
Jul 17 19:21:46.315820 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.315777    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"]
Jul 17 19:21:46.318644 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:21:46.318610    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4827f3d_1f61_4e76_ac0f_52a369bbc86e.slice/crio-772b9c0f91912bc0c9a5eb5348dc86691cb2ab6f16528c3eb8e03db96340bac9 WatchSource:0}: Error finding container 772b9c0f91912bc0c9a5eb5348dc86691cb2ab6f16528c3eb8e03db96340bac9: Status 404 returned error can't find the container with id 772b9c0f91912bc0c9a5eb5348dc86691cb2ab6f16528c3eb8e03db96340bac9
Jul 17 19:21:46.736168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.736132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:46.835309 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.835217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:46.897296 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.897264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" event={"ID":"c4827f3d-1f61-4e76-ac0f-52a369bbc86e","Type":"ContainerStarted","Data":"138b6217379a3aa124244082a538878e2f62b9af5f1c75208837f51d031d540d"}
Jul 17 19:21:46.897461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.897320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" event={"ID":"c4827f3d-1f61-4e76-ac0f-52a369bbc86e","Type":"ContainerStarted","Data":"772b9c0f91912bc0c9a5eb5348dc86691cb2ab6f16528c3eb8e03db96340bac9"}
Jul 17 19:21:46.911370 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.911326    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" podStartSLOduration=1.9113096729999999 podStartE2EDuration="1.911309673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:21:46.910431856 +0000 UTC m=+2915.821258629" watchObservedRunningTime="2026-07-17 19:21:46.911309673 +0000 UTC m=+2915.822136433"
Jul 17 19:21:46.915100 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:46.915078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:47.198246 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:47.198214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:47.616249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:47.616215    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:47.709552 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:47.709519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:47.833331 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:47.833294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:47.902816 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:47.902720    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerID="138b6217379a3aa124244082a538878e2f62b9af5f1c75208837f51d031d540d" exitCode=6
Jul 17 19:21:47.902816 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:47.902800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" event={"ID":"c4827f3d-1f61-4e76-ac0f-52a369bbc86e","Type":"ContainerDied","Data":"138b6217379a3aa124244082a538878e2f62b9af5f1c75208837f51d031d540d"}
Jul 17 19:21:47.903047 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:47.902997    2563 scope.go:117] "RemoveContainer" containerID="138b6217379a3aa124244082a538878e2f62b9af5f1c75208837f51d031d540d"
Jul 17 19:21:48.908592 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:48.908557    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerID="b7adf0f7e58dbaf5be3638896f34ef94d174e95b9e364c6c487ea0071aa1e500" exitCode=6
Jul 17 19:21:48.909000 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:48.908613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" event={"ID":"c4827f3d-1f61-4e76-ac0f-52a369bbc86e","Type":"ContainerDied","Data":"b7adf0f7e58dbaf5be3638896f34ef94d174e95b9e364c6c487ea0071aa1e500"}
Jul 17 19:21:48.909000 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:48.908659    2563 scope.go:117] "RemoveContainer" containerID="138b6217379a3aa124244082a538878e2f62b9af5f1c75208837f51d031d540d"
Jul 17 19:21:48.909000 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:48.908836    2563 scope.go:117] "RemoveContainer" containerID="b7adf0f7e58dbaf5be3638896f34ef94d174e95b9e364c6c487ea0071aa1e500"
Jul 17 19:21:48.909143 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:48.909070    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp_odh-ai-gateway-infra(c4827f3d-1f61-4e76-ac0f-52a369bbc86e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e"
Jul 17 19:21:49.158790 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:49.158758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:49.913977 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:49.913949    2563 scope.go:117] "RemoveContainer" containerID="b7adf0f7e58dbaf5be3638896f34ef94d174e95b9e364c6c487ea0071aa1e500"
Jul 17 19:21:49.914360 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:49.914179    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp_odh-ai-gateway-infra(c4827f3d-1f61-4e76-ac0f-52a369bbc86e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e"
Jul 17 19:21:50.095690 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.095654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:21:50.707883 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.707858    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:21:50.829808 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.829704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-credential-socket\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.829808 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.829754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-socket\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.829808 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.829807    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786c1c60-8786-40d0-9cce-69a15d8de50d-istiod-ca-cert\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.830168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.829838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-certs\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.830168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.829861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-token\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.830168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.829878    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x45b6\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-kube-api-access-x45b6\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.830168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.829916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-podinfo\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.830168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830049    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:21:50.830168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:21:50.830168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-data\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.830612 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:21:50.830612 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830195    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-envoy\") pod \"786c1c60-8786-40d0-9cce-69a15d8de50d\" (UID: \"786c1c60-8786-40d0-9cce-69a15d8de50d\") "
Jul 17 19:21:50.830612 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830461    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.830612 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830481    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.830612 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830496    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.830612 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786c1c60-8786-40d0-9cce-69a15d8de50d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:21:50.830985 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.830963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-data" (OuterVolumeSpecName: "istio-data") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:21:50.832361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.832333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:21:50.832477 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.832370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-kube-api-access-x45b6" (OuterVolumeSpecName: "kube-api-access-x45b6") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "kube-api-access-x45b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:21:50.832588 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.832568    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-token" (OuterVolumeSpecName: "istio-token") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:21:50.832652 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.832629    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "786c1c60-8786-40d0-9cce-69a15d8de50d" (UID: "786c1c60-8786-40d0-9cce-69a15d8de50d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:21:50.918841 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.918809    2563 generic.go:356] "Generic (PLEG): container finished" podID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerID="1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3" exitCode=0
Jul 17 19:21:50.919274 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.918862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" event={"ID":"786c1c60-8786-40d0-9cce-69a15d8de50d","Type":"ContainerDied","Data":"1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3"}
Jul 17 19:21:50.919274 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.918866    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"
Jul 17 19:21:50.919274 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.918906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54" event={"ID":"786c1c60-8786-40d0-9cce-69a15d8de50d","Type":"ContainerDied","Data":"a1424ea37858baddb271d088b428d1b7f61a1ca3081c842f8c523fc90e790e69"}
Jul 17 19:21:50.919274 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.918927    2563 scope.go:117] "RemoveContainer" containerID="1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3"
Jul 17 19:21:50.928218 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.928197    2563 scope.go:117] "RemoveContainer" containerID="c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0"
Jul 17 19:21:50.930960 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.930939    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786c1c60-8786-40d0-9cce-69a15d8de50d-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.931056 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.930962    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.931056 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.930975    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x45b6\" (UniqueName: \"kubernetes.io/projected/786c1c60-8786-40d0-9cce-69a15d8de50d-kube-api-access-x45b6\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.931056 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.930989    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.931056 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.931003    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.931056 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.931014    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786c1c60-8786-40d0-9cce-69a15d8de50d-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:21:50.935643 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.935626    2563 scope.go:117] "RemoveContainer" containerID="1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3"
Jul 17 19:21:50.935927 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:50.935907    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3\": container with ID starting with 1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3 not found: ID does not exist" containerID="1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3"
Jul 17 19:21:50.935976 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.935936    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3"} err="failed to get container status \"1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3\": rpc error: code = NotFound desc = could not find container \"1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3\": container with ID starting with 1efd02489d31be380289313baa813ae8639a49966a03847f8026f1e7829352e3 not found: ID does not exist"
Jul 17 19:21:50.935976 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.935953    2563 scope.go:117] "RemoveContainer" containerID="c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0"
Jul 17 19:21:50.936185 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:50.936166    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0\": container with ID starting with c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0 not found: ID does not exist" containerID="c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0"
Jul 17 19:21:50.936239 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.936193    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0"} err="failed to get container status \"c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0\": rpc error: code = NotFound desc = could not find container \"c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0\": container with ID starting with c4a57bd5cfa243c1f52dad132b1ec7d30ee7698d60d0dcc9c12d4e578ea3d4d0 not found: ID does not exist"
Jul 17 19:21:50.939991 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.939967    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"]
Jul 17 19:21:50.943985 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:50.943963    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-010d52ef-openshift-default-8f4d7bc59-zff54"]
Jul 17 19:21:51.193668 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.193637    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"]
Jul 17 19:21:51.193929 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.193918    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy"
Jul 17 19:21:51.193974 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.193931    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy"
Jul 17 19:21:51.193974 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.193945    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:51.193974 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.193950    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:51.194073 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.193993    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af1bacbf-3810-4aaa-92dd-841281deb47c" containerName="revoke-keys"
Jul 17 19:21:51.194073 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.194003    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy"
Jul 17 19:21:51.194073 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.194010    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="320724fb-bbeb-44a0-8110-a3aaccc6348d" containerName="revoke-keys"
Jul 17 19:21:51.194073 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.194017    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy"
Jul 17 19:21:51.198407 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.198389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"
Jul 17 19:21:51.204264 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.204181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"]
Jul 17 19:21:51.334136 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.334084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1de74e-5605-4a7d-9785-e27abb4fe601-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-wq72r\" (UID: \"cc1de74e-5605-4a7d-9785-e27abb4fe601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"
Jul 17 19:21:51.435249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.435207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1de74e-5605-4a7d-9785-e27abb4fe601-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-wq72r\" (UID: \"cc1de74e-5605-4a7d-9785-e27abb4fe601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"
Jul 17 19:21:51.435814 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.435794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1de74e-5605-4a7d-9785-e27abb4fe601-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-wq72r\" (UID: \"cc1de74e-5605-4a7d-9785-e27abb4fe601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"
Jul 17 19:21:51.510298 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.510214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"
Jul 17 19:21:51.639294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.639258    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"]
Jul 17 19:21:51.642079 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:21:51.642047    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1de74e_5605_4a7d_9785_e27abb4fe601.slice/crio-3dac06e8227a518f66661f2a5d8965a679896482741c94870df273774a82a292 WatchSource:0}: Error finding container 3dac06e8227a518f66661f2a5d8965a679896482741c94870df273774a82a292: Status 404 returned error can't find the container with id 3dac06e8227a518f66661f2a5d8965a679896482741c94870df273774a82a292
Jul 17 19:21:51.694556 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.694520    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" path="/var/lib/kubelet/pods/786c1c60-8786-40d0-9cce-69a15d8de50d/volumes"
Jul 17 19:21:51.924718 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.924686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" event={"ID":"cc1de74e-5605-4a7d-9785-e27abb4fe601","Type":"ContainerStarted","Data":"ded443579c104c484995130c3c7546159c953a17c442e9a2c7ceb252dd6cea2b"}
Jul 17 19:21:51.925145 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.924742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" event={"ID":"cc1de74e-5605-4a7d-9785-e27abb4fe601","Type":"ContainerStarted","Data":"3dac06e8227a518f66661f2a5d8965a679896482741c94870df273774a82a292"}
Jul 17 19:21:51.939423 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:51.939357    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" podStartSLOduration=0.939342962 podStartE2EDuration="939.342962ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:21:51.939339089 +0000 UTC m=+2920.850165868" watchObservedRunningTime="2026-07-17 19:21:51.939342962 +0000 UTC m=+2920.850169722"
Jul 17 19:21:52.931534 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:52.931497    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerID="ded443579c104c484995130c3c7546159c953a17c442e9a2c7ceb252dd6cea2b" exitCode=6
Jul 17 19:21:52.932009 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:52.931587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" event={"ID":"cc1de74e-5605-4a7d-9785-e27abb4fe601","Type":"ContainerDied","Data":"ded443579c104c484995130c3c7546159c953a17c442e9a2c7ceb252dd6cea2b"}
Jul 17 19:21:52.932009 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:52.931795    2563 scope.go:117] "RemoveContainer" containerID="ded443579c104c484995130c3c7546159c953a17c442e9a2c7ceb252dd6cea2b"
Jul 17 19:21:53.937949 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:53.937916    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerID="1637373f3e07bbfb67db687a43a0e7a8cbdd5b0ca5cda352d288979b6108848d" exitCode=6
Jul 17 19:21:53.938409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:53.937989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" event={"ID":"cc1de74e-5605-4a7d-9785-e27abb4fe601","Type":"ContainerDied","Data":"1637373f3e07bbfb67db687a43a0e7a8cbdd5b0ca5cda352d288979b6108848d"}
Jul 17 19:21:53.938409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:53.938044    2563 scope.go:117] "RemoveContainer" containerID="ded443579c104c484995130c3c7546159c953a17c442e9a2c7ceb252dd6cea2b"
Jul 17 19:21:53.938409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:53.938279    2563 scope.go:117] "RemoveContainer" containerID="1637373f3e07bbfb67db687a43a0e7a8cbdd5b0ca5cda352d288979b6108848d"
Jul 17 19:21:53.938556 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:53.938464    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-wq72r_odh-ai-gateway-infra(cc1de74e-5605-4a7d-9785-e27abb4fe601)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601"
Jul 17 19:21:54.944611 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:54.944581    2563 scope.go:117] "RemoveContainer" containerID="1637373f3e07bbfb67db687a43a0e7a8cbdd5b0ca5cda352d288979b6108848d"
Jul 17 19:21:54.945022 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:21:54.944752    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-wq72r_odh-ai-gateway-infra(cc1de74e-5605-4a7d-9785-e27abb4fe601)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601"
Jul 17 19:21:56.034342 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:56.034289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:56.134350 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:56.134318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:21:56.698226 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:21:56.698184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:03.749187 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:03.749155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:03.849803 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:03.849764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:03.949365 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:03.949331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:04.078092 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.078012    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"]
Jul 17 19:22:04.078335 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.078322    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy"
Jul 17 19:22:04.078379 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.078336    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="786c1c60-8786-40d0-9cce-69a15d8de50d" containerName="istio-proxy"
Jul 17 19:22:04.082475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.082457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"
Jul 17 19:22:04.096412 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.096365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"]
Jul 17 19:22:04.142995 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.142962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2034b1a5-b0e3-4cad-bde7-2aff0501f461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv\" (UID: \"2034b1a5-b0e3-4cad-bde7-2aff0501f461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"
Jul 17 19:22:04.244431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.244398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2034b1a5-b0e3-4cad-bde7-2aff0501f461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv\" (UID: \"2034b1a5-b0e3-4cad-bde7-2aff0501f461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"
Jul 17 19:22:04.245066 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.245047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2034b1a5-b0e3-4cad-bde7-2aff0501f461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv\" (UID: \"2034b1a5-b0e3-4cad-bde7-2aff0501f461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"
Jul 17 19:22:04.392050 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.392015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"
Jul 17 19:22:04.522263 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.522173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"]
Jul 17 19:22:04.524946 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:22:04.524915    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2034b1a5_b0e3_4cad_bde7_2aff0501f461.slice/crio-921e465a71463312843ff20d7dfd433b0c0dc43d883a39b0b93c225d1cd90baa WatchSource:0}: Error finding container 921e465a71463312843ff20d7dfd433b0c0dc43d883a39b0b93c225d1cd90baa: Status 404 returned error can't find the container with id 921e465a71463312843ff20d7dfd433b0c0dc43d883a39b0b93c225d1cd90baa
Jul 17 19:22:04.689503 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.689425    2563 scope.go:117] "RemoveContainer" containerID="b7adf0f7e58dbaf5be3638896f34ef94d174e95b9e364c6c487ea0071aa1e500"
Jul 17 19:22:04.799877 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.799843    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:22:04.991031 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.991000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" event={"ID":"2034b1a5-b0e3-4cad-bde7-2aff0501f461","Type":"ContainerStarted","Data":"46f44b843752a7934ea6828ed19908d57f303371c0a4d2e11c955a19c9e940e7"}
Jul 17 19:22:04.991197 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.991059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" event={"ID":"2034b1a5-b0e3-4cad-bde7-2aff0501f461","Type":"ContainerStarted","Data":"921e465a71463312843ff20d7dfd433b0c0dc43d883a39b0b93c225d1cd90baa"}
Jul 17 19:22:04.992630 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:04.992610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" event={"ID":"c4827f3d-1f61-4e76-ac0f-52a369bbc86e","Type":"ContainerStarted","Data":"8e6a680f323aec4455a5be30b0afa12526d0569e3f46d4e734e23012a9d0533e"}
Jul 17 19:22:05.010831 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:05.010781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" podStartSLOduration=1.010767476 podStartE2EDuration="1.010767476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:05.010003656 +0000 UTC m=+2933.920830435" watchObservedRunningTime="2026-07-17 19:22:05.010767476 +0000 UTC m=+2933.921594237"
Jul 17 19:22:05.998401 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:05.998362    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerID="8e6a680f323aec4455a5be30b0afa12526d0569e3f46d4e734e23012a9d0533e" exitCode=6
Jul 17 19:22:05.998832 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:05.998437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" event={"ID":"c4827f3d-1f61-4e76-ac0f-52a369bbc86e","Type":"ContainerDied","Data":"8e6a680f323aec4455a5be30b0afa12526d0569e3f46d4e734e23012a9d0533e"}
Jul 17 19:22:05.998832 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:05.998501    2563 scope.go:117] "RemoveContainer" containerID="b7adf0f7e58dbaf5be3638896f34ef94d174e95b9e364c6c487ea0071aa1e500"
Jul 17 19:22:05.998832 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:05.998816    2563 scope.go:117] "RemoveContainer" containerID="8e6a680f323aec4455a5be30b0afa12526d0569e3f46d4e734e23012a9d0533e"
Jul 17 19:22:05.999086 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:05.999047    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp_odh-ai-gateway-infra(c4827f3d-1f61-4e76-ac0f-52a369bbc86e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e"
Jul 17 19:22:06.000046 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:06.000023    2563 generic.go:356] "Generic (PLEG): container finished" podID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerID="46f44b843752a7934ea6828ed19908d57f303371c0a4d2e11c955a19c9e940e7" exitCode=6
Jul 17 19:22:06.000170 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:06.000143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" event={"ID":"2034b1a5-b0e3-4cad-bde7-2aff0501f461","Type":"ContainerDied","Data":"46f44b843752a7934ea6828ed19908d57f303371c0a4d2e11c955a19c9e940e7"}
Jul 17 19:22:06.000272 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:06.000260    2563 scope.go:117] "RemoveContainer" containerID="46f44b843752a7934ea6828ed19908d57f303371c0a4d2e11c955a19c9e940e7"
Jul 17 19:22:06.051830 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:06.051796    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"]
Jul 17 19:22:06.689092 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:06.689062    2563 scope.go:117] "RemoveContainer" containerID="1637373f3e07bbfb67db687a43a0e7a8cbdd5b0ca5cda352d288979b6108848d"
Jul 17 19:22:07.007611 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.007577    2563 generic.go:356] "Generic (PLEG): container finished" podID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerID="b5a1dc736b846363adb32aa12d060f8e580b60e2678b8be48161afabc46783ab" exitCode=6
Jul 17 19:22:07.008038 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.007658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" event={"ID":"2034b1a5-b0e3-4cad-bde7-2aff0501f461","Type":"ContainerDied","Data":"b5a1dc736b846363adb32aa12d060f8e580b60e2678b8be48161afabc46783ab"}
Jul 17 19:22:07.008038 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.007726    2563 scope.go:117] "RemoveContainer" containerID="46f44b843752a7934ea6828ed19908d57f303371c0a4d2e11c955a19c9e940e7"
Jul 17 19:22:07.008038 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.007855    2563 scope.go:117] "RemoveContainer" containerID="b5a1dc736b846363adb32aa12d060f8e580b60e2678b8be48161afabc46783ab"
Jul 17 19:22:07.008254 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:07.008092    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv_odh-ai-gateway-infra(2034b1a5-b0e3-4cad-bde7-2aff0501f461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461"
Jul 17 19:22:07.009728 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.009695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" event={"ID":"cc1de74e-5605-4a7d-9785-e27abb4fe601","Type":"ContainerStarted","Data":"0dd32069cb2775ba1f79021dfe60cb6d11068331ba56b1a4b6e9a9287481fc83"}
Jul 17 19:22:07.138981 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.138955    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"
Jul 17 19:22:07.166667 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.166593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4827f3d-1f61-4e76-ac0f-52a369bbc86e-maas-api-service-ca\") pod \"c4827f3d-1f61-4e76-ac0f-52a369bbc86e\" (UID: \"c4827f3d-1f61-4e76-ac0f-52a369bbc86e\") "
Jul 17 19:22:07.166919 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.166899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4827f3d-1f61-4e76-ac0f-52a369bbc86e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4827f3d-1f61-4e76-ac0f-52a369bbc86e" (UID: "c4827f3d-1f61-4e76-ac0f-52a369bbc86e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:22:07.267828 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:07.267795    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4827f3d-1f61-4e76-ac0f-52a369bbc86e-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:22:07.398901 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:07.398871    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1de74e_5605_4a7d_9785_e27abb4fe601.slice/crio-0dd32069cb2775ba1f79021dfe60cb6d11068331ba56b1a4b6e9a9287481fc83.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 19:22:08.015164 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.015053    2563 scope.go:117] "RemoveContainer" containerID="b5a1dc736b846363adb32aa12d060f8e580b60e2678b8be48161afabc46783ab"
Jul 17 19:22:08.015549 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:08.015302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv_odh-ai-gateway-infra(2034b1a5-b0e3-4cad-bde7-2aff0501f461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461"
Jul 17 19:22:08.016751 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.016725    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerID="0dd32069cb2775ba1f79021dfe60cb6d11068331ba56b1a4b6e9a9287481fc83" exitCode=6
Jul 17 19:22:08.016878 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.016804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" event={"ID":"cc1de74e-5605-4a7d-9785-e27abb4fe601","Type":"ContainerDied","Data":"0dd32069cb2775ba1f79021dfe60cb6d11068331ba56b1a4b6e9a9287481fc83"}
Jul 17 19:22:08.016878 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.016850    2563 scope.go:117] "RemoveContainer" containerID="1637373f3e07bbfb67db687a43a0e7a8cbdd5b0ca5cda352d288979b6108848d"
Jul 17 19:22:08.017112 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.017098    2563 scope.go:117] "RemoveContainer" containerID="0dd32069cb2775ba1f79021dfe60cb6d11068331ba56b1a4b6e9a9287481fc83"
Jul 17 19:22:08.017347 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:08.017307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-wq72r_odh-ai-gateway-infra(cc1de74e-5605-4a7d-9785-e27abb4fe601)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601"
Jul 17 19:22:08.018034 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.018015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp" event={"ID":"c4827f3d-1f61-4e76-ac0f-52a369bbc86e","Type":"ContainerDied","Data":"772b9c0f91912bc0c9a5eb5348dc86691cb2ab6f16528c3eb8e03db96340bac9"}
Jul 17 19:22:08.018085 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.018032    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"
Jul 17 19:22:08.027419 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.027399    2563 scope.go:117] "RemoveContainer" containerID="8e6a680f323aec4455a5be30b0afa12526d0569e3f46d4e734e23012a9d0533e"
Jul 17 19:22:08.047003 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.046929    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"]
Jul 17 19:22:08.055092 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.055061    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"]
Jul 17 19:22:08.056517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:08.056497    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-97bsp"]
Jul 17 19:22:09.159196 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:09.159173    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"
Jul 17 19:22:09.183458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:09.183429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1de74e-5605-4a7d-9785-e27abb4fe601-maas-api-service-ca\") pod \"cc1de74e-5605-4a7d-9785-e27abb4fe601\" (UID: \"cc1de74e-5605-4a7d-9785-e27abb4fe601\") "
Jul 17 19:22:09.183852 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:09.183808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1de74e-5605-4a7d-9785-e27abb4fe601-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc1de74e-5605-4a7d-9785-e27abb4fe601" (UID: "cc1de74e-5605-4a7d-9785-e27abb4fe601"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:22:09.284136 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:09.284031    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1de74e-5605-4a7d-9785-e27abb4fe601-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:22:09.693590 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:09.693553    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" path="/var/lib/kubelet/pods/c4827f3d-1f61-4e76-ac0f-52a369bbc86e/volumes"
Jul 17 19:22:10.029578 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:10.029487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r" event={"ID":"cc1de74e-5605-4a7d-9785-e27abb4fe601","Type":"ContainerDied","Data":"3dac06e8227a518f66661f2a5d8965a679896482741c94870df273774a82a292"}
Jul 17 19:22:10.029578 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:10.029515    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"
Jul 17 19:22:10.029578 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:10.029551    2563 scope.go:117] "RemoveContainer" containerID="0dd32069cb2775ba1f79021dfe60cb6d11068331ba56b1a4b6e9a9287481fc83"
Jul 17 19:22:10.047721 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:10.047690    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"]
Jul 17 19:22:10.053450 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:10.053420    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-wq72r"]
Jul 17 19:22:11.694583 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:11.694547    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" path="/var/lib/kubelet/pods/cc1de74e-5605-4a7d-9785-e27abb4fe601/volumes"
Jul 17 19:22:14.380433 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:14.380392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:14.728721 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:14.728631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:19.690010 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:19.689980    2563 scope.go:117] "RemoveContainer" containerID="b5a1dc736b846363adb32aa12d060f8e580b60e2678b8be48161afabc46783ab"
Jul 17 19:22:20.069881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:20.069795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" event={"ID":"2034b1a5-b0e3-4cad-bde7-2aff0501f461","Type":"ContainerStarted","Data":"b39d3e5e167d44378eef51a9317fbd666033e78318f7176a3a58e4637feb36b4"}
Jul 17 19:22:21.075597 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:21.075564    2563 generic.go:356] "Generic (PLEG): container finished" podID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerID="b39d3e5e167d44378eef51a9317fbd666033e78318f7176a3a58e4637feb36b4" exitCode=6
Jul 17 19:22:21.076054 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:21.075614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" event={"ID":"2034b1a5-b0e3-4cad-bde7-2aff0501f461","Type":"ContainerDied","Data":"b39d3e5e167d44378eef51a9317fbd666033e78318f7176a3a58e4637feb36b4"}
Jul 17 19:22:21.076054 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:21.075666    2563 scope.go:117] "RemoveContainer" containerID="b5a1dc736b846363adb32aa12d060f8e580b60e2678b8be48161afabc46783ab"
Jul 17 19:22:21.076054 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:21.076018    2563 scope.go:117] "RemoveContainer" containerID="b39d3e5e167d44378eef51a9317fbd666033e78318f7176a3a58e4637feb36b4"
Jul 17 19:22:21.076306 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:21.076284    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv_odh-ai-gateway-infra(2034b1a5-b0e3-4cad-bde7-2aff0501f461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461"
Jul 17 19:22:21.096859 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:21.096829    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"]
Jul 17 19:22:22.220979 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:22.220949    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"
Jul 17 19:22:22.297880 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:22.297847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2034b1a5-b0e3-4cad-bde7-2aff0501f461-maas-api-service-ca\") pod \"2034b1a5-b0e3-4cad-bde7-2aff0501f461\" (UID: \"2034b1a5-b0e3-4cad-bde7-2aff0501f461\") "
Jul 17 19:22:22.298227 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:22.298205    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2034b1a5-b0e3-4cad-bde7-2aff0501f461-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2034b1a5-b0e3-4cad-bde7-2aff0501f461" (UID: "2034b1a5-b0e3-4cad-bde7-2aff0501f461"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:22:22.399335 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:22.399298    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2034b1a5-b0e3-4cad-bde7-2aff0501f461-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:22:23.086071 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:23.086039    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"
Jul 17 19:22:23.086071 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:23.086049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv" event={"ID":"2034b1a5-b0e3-4cad-bde7-2aff0501f461","Type":"ContainerDied","Data":"921e465a71463312843ff20d7dfd433b0c0dc43d883a39b0b93c225d1cd90baa"}
Jul 17 19:22:23.086325 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:23.086109    2563 scope.go:117] "RemoveContainer" containerID="b39d3e5e167d44378eef51a9317fbd666033e78318f7176a3a58e4637feb36b4"
Jul 17 19:22:23.105741 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:23.105712    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"]
Jul 17 19:22:23.109503 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:23.109480    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-dwfpv"]
Jul 17 19:22:23.694002 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:23.693972    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" path="/var/lib/kubelet/pods/2034b1a5-b0e3-4cad-bde7-2aff0501f461/volumes"
Jul 17 19:22:32.958480 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958394    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"]
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958659    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958670    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958676    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958682    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958688    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958695    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958700    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958706    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958717    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958722    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958728    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958733    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958745    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958752    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958758    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958797    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958803    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958817    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958823    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958829    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:32.958875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.958837    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:32.963071 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.963053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"
Jul 17 19:22:32.965386 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.965363    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:22:32.965527 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.965416    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:22:32.968541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:32.968513    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"]
Jul 17 19:22:33.087160 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:33.087099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b8d985-8e28-4ead-940b-182ed6054a20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v\" (UID: \"97b8d985-8e28-4ead-940b-182ed6054a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"
Jul 17 19:22:33.188028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:33.187987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b8d985-8e28-4ead-940b-182ed6054a20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v\" (UID: \"97b8d985-8e28-4ead-940b-182ed6054a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"
Jul 17 19:22:33.188630 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:33.188610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b8d985-8e28-4ead-940b-182ed6054a20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v\" (UID: \"97b8d985-8e28-4ead-940b-182ed6054a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"
Jul 17 19:22:33.274836 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:33.274746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"
Jul 17 19:22:33.405651 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:33.405623    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"]
Jul 17 19:22:33.408056 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:22:33.408027    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b8d985_8e28_4ead_940b_182ed6054a20.slice/crio-32c22d95594f970583dba5016c2fc61bd16b4501eb3379b62cd5316fabce4d33 WatchSource:0}: Error finding container 32c22d95594f970583dba5016c2fc61bd16b4501eb3379b62cd5316fabce4d33: Status 404 returned error can't find the container with id 32c22d95594f970583dba5016c2fc61bd16b4501eb3379b62cd5316fabce4d33
Jul 17 19:22:34.132053 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:34.132021    2563 generic.go:356] "Generic (PLEG): container finished" podID="97b8d985-8e28-4ead-940b-182ed6054a20" containerID="c72dbfa2dc5f8c99034ca2c5d782c29699e86618009f809caa0897dd8a818f3d" exitCode=6
Jul 17 19:22:34.132458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:34.132074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" event={"ID":"97b8d985-8e28-4ead-940b-182ed6054a20","Type":"ContainerDied","Data":"c72dbfa2dc5f8c99034ca2c5d782c29699e86618009f809caa0897dd8a818f3d"}
Jul 17 19:22:34.132458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:34.132114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" event={"ID":"97b8d985-8e28-4ead-940b-182ed6054a20","Type":"ContainerStarted","Data":"32c22d95594f970583dba5016c2fc61bd16b4501eb3379b62cd5316fabce4d33"}
Jul 17 19:22:34.132458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:34.132336    2563 scope.go:117] "RemoveContainer" containerID="c72dbfa2dc5f8c99034ca2c5d782c29699e86618009f809caa0897dd8a818f3d"
Jul 17 19:22:35.138713 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:35.138680    2563 generic.go:356] "Generic (PLEG): container finished" podID="97b8d985-8e28-4ead-940b-182ed6054a20" containerID="cbf68b7214946397fd5ff39b7ccd363bf43bbc5b3eef48a5efbb16220a33f1f2" exitCode=6
Jul 17 19:22:35.139146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:35.138765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" event={"ID":"97b8d985-8e28-4ead-940b-182ed6054a20","Type":"ContainerDied","Data":"cbf68b7214946397fd5ff39b7ccd363bf43bbc5b3eef48a5efbb16220a33f1f2"}
Jul 17 19:22:35.139146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:35.138827    2563 scope.go:117] "RemoveContainer" containerID="c72dbfa2dc5f8c99034ca2c5d782c29699e86618009f809caa0897dd8a818f3d"
Jul 17 19:22:35.139146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:35.139102    2563 scope.go:117] "RemoveContainer" containerID="cbf68b7214946397fd5ff39b7ccd363bf43bbc5b3eef48a5efbb16220a33f1f2"
Jul 17 19:22:35.139355 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:35.139335    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v_odh-ai-gateway-infra(97b8d985-8e28-4ead-940b-182ed6054a20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" podUID="97b8d985-8e28-4ead-940b-182ed6054a20"
Jul 17 19:22:35.672358 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:35.672313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:36.143898 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:36.143874    2563 scope.go:117] "RemoveContainer" containerID="cbf68b7214946397fd5ff39b7ccd363bf43bbc5b3eef48a5efbb16220a33f1f2"
Jul 17 19:22:36.144379 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:36.144105    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v_odh-ai-gateway-infra(97b8d985-8e28-4ead-940b-182ed6054a20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" podUID="97b8d985-8e28-4ead-940b-182ed6054a20"
Jul 17 19:22:41.237546 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.237506    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"]
Jul 17 19:22:41.237965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.237785    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:41.237965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.237796    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4827f3d-1f61-4e76-ac0f-52a369bbc86e" containerName="revoke-keys"
Jul 17 19:22:41.237965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.237849    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc1de74e-5605-4a7d-9785-e27abb4fe601" containerName="revoke-keys"
Jul 17 19:22:41.237965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.237860    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2034b1a5-b0e3-4cad-bde7-2aff0501f461" containerName="revoke-keys"
Jul 17 19:22:41.242084 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.242067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"
Jul 17 19:22:41.248028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.247994    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"]
Jul 17 19:22:41.364847 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.364801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc4e99-80f8-4c4b-987b-fe8250f98f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm\" (UID: \"a9fc4e99-80f8-4c4b-987b-fe8250f98f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"
Jul 17 19:22:41.465639 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.465593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc4e99-80f8-4c4b-987b-fe8250f98f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm\" (UID: \"a9fc4e99-80f8-4c4b-987b-fe8250f98f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"
Jul 17 19:22:41.466196 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.466175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc4e99-80f8-4c4b-987b-fe8250f98f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm\" (UID: \"a9fc4e99-80f8-4c4b-987b-fe8250f98f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"
Jul 17 19:22:41.501298 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.501212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:41.554202 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.554169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"
Jul 17 19:22:41.600095 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.600027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:41.685902 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.685866    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"]
Jul 17 19:22:41.689039 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:22:41.689011    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fc4e99_80f8_4c4b_987b_fe8250f98f07.slice/crio-4e1db05fee1a25ad11afd04a4934e3261983f8415eb528775d5a57e8e1fee03f WatchSource:0}: Error finding container 4e1db05fee1a25ad11afd04a4934e3261983f8415eb528775d5a57e8e1fee03f: Status 404 returned error can't find the container with id 4e1db05fee1a25ad11afd04a4934e3261983f8415eb528775d5a57e8e1fee03f
Jul 17 19:22:41.852072 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:41.852024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:42.134360 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.134272    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"]
Jul 17 19:22:42.137505 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.137484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.140006 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.139983    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4aee7c-openshift-default-dockercfg-r7krg\""
Jul 17 19:22:42.149375 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.149352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"]
Jul 17 19:22:42.161095 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.161059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:42.177869 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.177824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" event={"ID":"a9fc4e99-80f8-4c4b-987b-fe8250f98f07","Type":"ContainerStarted","Data":"b6900eeaaa378def7c5b07652aa183c7385dba9d174162ff4039907649b79c46"}
Jul 17 19:22:42.178026 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.177923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" event={"ID":"a9fc4e99-80f8-4c4b-987b-fe8250f98f07","Type":"ContainerStarted","Data":"4e1db05fee1a25ad11afd04a4934e3261983f8415eb528775d5a57e8e1fee03f"}
Jul 17 19:22:42.199375 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.199309    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" podStartSLOduration=1.199289211 podStartE2EDuration="1.199289211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:42.198455522 +0000 UTC m=+2971.109282329" watchObservedRunningTime="2026-07-17 19:22:42.199289211 +0000 UTC m=+2971.110115965"
Jul 17 19:22:42.261716 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.261672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:42.273932 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.273896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-podinfo\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.274101 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.273939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-data\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.274101 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.273963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qrt\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-kube-api-access-q4qrt\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.274101 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.274050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-certs\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.274249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.274105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-envoy\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.274249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.274147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istiod-ca-cert\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.274249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.274172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-socket\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.274249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.274203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-credential-socket\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.274249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.274222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-token\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375262 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-certs\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375473 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-envoy\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375473 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istiod-ca-cert\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375473 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-socket\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375473 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-credential-socket\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375473 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-token\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375473 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-podinfo\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375778 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-data\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375778 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qrt\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-kube-api-access-q4qrt\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.375925 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-certs\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.376068 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.375931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-data\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.376068 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.376000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-socket\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.376251 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.376231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istiod-ca-cert\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.376493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.376371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-credential-socket\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.378162 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.378142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-envoy\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.378371 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.378349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-podinfo\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.383707 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.383685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-token\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.383987 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.383964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qrt\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-kube-api-access-q4qrt\") pod \"e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") " pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.448086 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.448050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:42.590892 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.590864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"]
Jul 17 19:22:42.593028 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:22:42.592999    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ca82f7_e0d9_472f_b3ec_39e74a2afc30.slice/crio-ce87625277d15b9d26d4c1760f14a6ccfd7ef804b0a91c099e90ae7dcbeffb77 WatchSource:0}: Error finding container ce87625277d15b9d26d4c1760f14a6ccfd7ef804b0a91c099e90ae7dcbeffb77: Status 404 returned error can't find the container with id ce87625277d15b9d26d4c1760f14a6ccfd7ef804b0a91c099e90ae7dcbeffb77
Jul 17 19:22:42.595260 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.595227    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:22:42.595381 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.595305    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:22:42.595381 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:42.595350    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:22:43.184211 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:43.184174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" event={"ID":"82ca82f7-e0d9-472f-b3ec-39e74a2afc30","Type":"ContainerStarted","Data":"e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8"}
Jul 17 19:22:43.184374 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:43.184234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" event={"ID":"82ca82f7-e0d9-472f-b3ec-39e74a2afc30","Type":"ContainerStarted","Data":"ce87625277d15b9d26d4c1760f14a6ccfd7ef804b0a91c099e90ae7dcbeffb77"}
Jul 17 19:22:43.185662 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:43.185636    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerID="b6900eeaaa378def7c5b07652aa183c7385dba9d174162ff4039907649b79c46" exitCode=6
Jul 17 19:22:43.185783 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:43.185713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" event={"ID":"a9fc4e99-80f8-4c4b-987b-fe8250f98f07","Type":"ContainerDied","Data":"b6900eeaaa378def7c5b07652aa183c7385dba9d174162ff4039907649b79c46"}
Jul 17 19:22:43.185850 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:43.185833    2563 scope.go:117] "RemoveContainer" containerID="b6900eeaaa378def7c5b07652aa183c7385dba9d174162ff4039907649b79c46"
Jul 17 19:22:43.205801 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:43.205741    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" podStartSLOduration=1.205726222 podStartE2EDuration="1.205726222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:43.203339733 +0000 UTC m=+2972.114166521" watchObservedRunningTime="2026-07-17 19:22:43.205726222 +0000 UTC m=+2972.116552982"
Jul 17 19:22:43.449282 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:43.449174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:43.455001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:43.454968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:44.191780 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:44.191746    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerID="1ddbb25b96fadfd7adabc524c997ff3568d601a4d18ad91e1995927c8d448490" exitCode=6
Jul 17 19:22:44.191957 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:44.191829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" event={"ID":"a9fc4e99-80f8-4c4b-987b-fe8250f98f07","Type":"ContainerDied","Data":"1ddbb25b96fadfd7adabc524c997ff3568d601a4d18ad91e1995927c8d448490"}
Jul 17 19:22:44.191957 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:44.191898    2563 scope.go:117] "RemoveContainer" containerID="b6900eeaaa378def7c5b07652aa183c7385dba9d174162ff4039907649b79c46"
Jul 17 19:22:44.192095 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:44.191994    2563 scope.go:117] "RemoveContainer" containerID="1ddbb25b96fadfd7adabc524c997ff3568d601a4d18ad91e1995927c8d448490"
Jul 17 19:22:44.192251 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:44.192227    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm_odh-ai-gateway-infra(a9fc4e99-80f8-4c4b-987b-fe8250f98f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07"
Jul 17 19:22:44.192251 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:44.192246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:44.193474 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:44.193459    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:22:45.195307 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.195272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:45.198232 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.198211    2563 scope.go:117] "RemoveContainer" containerID="1ddbb25b96fadfd7adabc524c997ff3568d601a4d18ad91e1995927c8d448490"
Jul 17 19:22:45.198424 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:45.198403    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm_odh-ai-gateway-infra(a9fc4e99-80f8-4c4b-987b-fe8250f98f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07"
Jul 17 19:22:45.286583 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.286554    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"]
Jul 17 19:22:45.290965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.290948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.294857 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.294835    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4aee7c-serving-cert\""
Jul 17 19:22:45.294970 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.294856    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:22:45.295476 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.295451    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:22:45.299392 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.299371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97x5\" (UniqueName: \"kubernetes.io/projected/c1a29993-4df1-4262-96b3-a998db8587cf-kube-api-access-d97x5\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.299508 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.299434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.299580 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.299515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a29993-4df1-4262-96b3-a998db8587cf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.307063 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.307039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"]
Jul 17 19:22:45.343597 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.343563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:45.400603 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.400567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.400762 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.400617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a29993-4df1-4262-96b3-a998db8587cf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.400762 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.400697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97x5\" (UniqueName: \"kubernetes.io/projected/c1a29993-4df1-4262-96b3-a998db8587cf-kube-api-access-d97x5\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.400762 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:45.400727    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-serving-cert: secret "maas-api-e2e-inf-a-4aee7c-serving-cert" not found
Jul 17 19:22:45.400900 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:45.400807    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls podName:c1a29993-4df1-4262-96b3-a998db8587cf nodeName:}" failed. No retries permitted until 2026-07-17 19:22:45.900777835 +0000 UTC m=+2974.811604593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls") pod "maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk" (UID: "c1a29993-4df1-4262-96b3-a998db8587cf") : secret "maas-api-e2e-inf-a-4aee7c-serving-cert" not found
Jul 17 19:22:45.401356 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.401332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a29993-4df1-4262-96b3-a998db8587cf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.409864 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.409836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97x5\" (UniqueName: \"kubernetes.io/projected/c1a29993-4df1-4262-96b3-a998db8587cf-kube-api-access-d97x5\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.432220 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.432191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:45.745830 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.745797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:45.850991 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.850960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:45.905964 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.905925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.908489 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.908461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls\") pod \"maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:45.926414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:45.926379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:46.201270 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:46.201240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:46.337677 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:46.337587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"]
Jul 17 19:22:46.339860 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:22:46.339833    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a29993_4df1_4262_96b3_a998db8587cf.slice/crio-718063d5774e775b9b49727cf43104cdc4ecf14f9af098dfb2775aa562ee8385 WatchSource:0}: Error finding container 718063d5774e775b9b49727cf43104cdc4ecf14f9af098dfb2775aa562ee8385: Status 404 returned error can't find the container with id 718063d5774e775b9b49727cf43104cdc4ecf14f9af098dfb2775aa562ee8385
Jul 17 19:22:47.209358 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:47.209258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk" event={"ID":"c1a29993-4df1-4262-96b3-a998db8587cf","Type":"ContainerStarted","Data":"08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb"}
Jul 17 19:22:47.209820 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:47.209363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk" event={"ID":"c1a29993-4df1-4262-96b3-a998db8587cf","Type":"ContainerStarted","Data":"718063d5774e775b9b49727cf43104cdc4ecf14f9af098dfb2775aa562ee8385"}
Jul 17 19:22:47.209820 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:47.209483    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:47.229799 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:47.229749    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk" podStartSLOduration=1.705657526 podStartE2EDuration="2.229731232s" totalImagesPullingTime="524.073706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:47.227930579 +0000 UTC m=+2976.138757365" watchObservedRunningTime="2026-07-17 19:22:47.229731232 +0000 UTC m=+2976.140558027"
Jul 17 19:22:48.689795 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:48.689766    2563 scope.go:117] "RemoveContainer" containerID="cbf68b7214946397fd5ff39b7ccd363bf43bbc5b3eef48a5efbb16220a33f1f2"
Jul 17 19:22:49.218897 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:49.218863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" event={"ID":"97b8d985-8e28-4ead-940b-182ed6054a20","Type":"ContainerStarted","Data":"99ff9af713040afcaa1d68144f887f5d404baf1f2795216629d4e837acfcc23b"}
Jul 17 19:22:49.236556 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:49.236500    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" podStartSLOduration=17.236482159 podStartE2EDuration="17.236482159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:49.235716725 +0000 UTC m=+2978.146543520" watchObservedRunningTime="2026-07-17 19:22:49.236482159 +0000 UTC m=+2978.147308915"
Jul 17 19:22:49.629638 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:49.629601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:49.950951 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:49.950862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:50.226224 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:50.226147    2563 generic.go:356] "Generic (PLEG): container finished" podID="97b8d985-8e28-4ead-940b-182ed6054a20" containerID="99ff9af713040afcaa1d68144f887f5d404baf1f2795216629d4e837acfcc23b" exitCode=6
Jul 17 19:22:50.226361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:50.226226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" event={"ID":"97b8d985-8e28-4ead-940b-182ed6054a20","Type":"ContainerDied","Data":"99ff9af713040afcaa1d68144f887f5d404baf1f2795216629d4e837acfcc23b"}
Jul 17 19:22:50.226361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:50.226305    2563 scope.go:117] "RemoveContainer" containerID="cbf68b7214946397fd5ff39b7ccd363bf43bbc5b3eef48a5efbb16220a33f1f2"
Jul 17 19:22:50.226604 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:50.226583    2563 scope.go:117] "RemoveContainer" containerID="99ff9af713040afcaa1d68144f887f5d404baf1f2795216629d4e837acfcc23b"
Jul 17 19:22:50.226820 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:50.226802    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v_odh-ai-gateway-infra(97b8d985-8e28-4ead-940b-182ed6054a20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" podUID="97b8d985-8e28-4ead-940b-182ed6054a20"
Jul 17 19:22:50.258287 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:50.258254    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"]
Jul 17 19:22:50.272390 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:50.272355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:50.452541 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:50.452501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:51.360241 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:51.360220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"
Jul 17 19:22:51.455995 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:51.455961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b8d985-8e28-4ead-940b-182ed6054a20-maas-api-service-ca\") pod \"97b8d985-8e28-4ead-940b-182ed6054a20\" (UID: \"97b8d985-8e28-4ead-940b-182ed6054a20\") "
Jul 17 19:22:51.456362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:51.456340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b8d985-8e28-4ead-940b-182ed6054a20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97b8d985-8e28-4ead-940b-182ed6054a20" (UID: "97b8d985-8e28-4ead-940b-182ed6054a20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:22:51.556576 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:51.556490    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b8d985-8e28-4ead-940b-182ed6054a20-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:22:52.237602 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:52.237560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v" event={"ID":"97b8d985-8e28-4ead-940b-182ed6054a20","Type":"ContainerDied","Data":"32c22d95594f970583dba5016c2fc61bd16b4501eb3379b62cd5316fabce4d33"}
Jul 17 19:22:52.237793 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:52.237573    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"
Jul 17 19:22:52.237793 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:52.237655    2563 scope.go:117] "RemoveContainer" containerID="99ff9af713040afcaa1d68144f887f5d404baf1f2795216629d4e837acfcc23b"
Jul 17 19:22:52.253732 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:52.253706    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"]
Jul 17 19:22:52.259008 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:52.258987    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6xz6v"]
Jul 17 19:22:53.211315 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.211277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:53.219199 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.219171    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:22:53.240296 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.240265    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:22:53.434027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.433989    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"]
Jul 17 19:22:53.434513 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.434492    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:22:53.434566 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.434518    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:22:53.434566 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.434553    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:22:53.434566 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.434562    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:22:53.434668 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.434651    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:22:53.434668 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.434663    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:22:53.434732 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.434674    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:22:53.439459 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.439438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.442201 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.442176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-15f29d-serving-cert\""
Jul 17 19:22:53.452449 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.452422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"]
Jul 17 19:22:53.472542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.472434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.472542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.472523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbqj\" (UniqueName: \"kubernetes.io/projected/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-kube-api-access-zxbqj\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.472755 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.472598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-maas-api-tls\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.574149 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.574101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-maas-api-tls\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.574342 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.574196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.574342 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.574226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbqj\" (UniqueName: \"kubernetes.io/projected/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-kube-api-access-zxbqj\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.574887 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.574864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.576803 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.576779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-maas-api-tls\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.584881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.584859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbqj\" (UniqueName: \"kubernetes.io/projected/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-kube-api-access-zxbqj\") pod \"maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.694277 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.694240    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" path="/var/lib/kubelet/pods/97b8d985-8e28-4ead-940b-182ed6054a20/volumes"
Jul 17 19:22:53.753223 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.753104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:53.892919 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:53.892802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"]
Jul 17 19:22:53.895540 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:22:53.895512    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a92362_0a05_4a5a_91c4_f3a2ad6be4fd.slice/crio-3c41e07e854563bc600cbbacc0fdfa4c05e52c58803b814990c96ceea52d5fc5 WatchSource:0}: Error finding container 3c41e07e854563bc600cbbacc0fdfa4c05e52c58803b814990c96ceea52d5fc5: Status 404 returned error can't find the container with id 3c41e07e854563bc600cbbacc0fdfa4c05e52c58803b814990c96ceea52d5fc5
Jul 17 19:22:54.248539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:54.248507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2" event={"ID":"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd","Type":"ContainerStarted","Data":"3c41e07e854563bc600cbbacc0fdfa4c05e52c58803b814990c96ceea52d5fc5"}
Jul 17 19:22:54.457992 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:54.457957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:55.254976 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:55.254938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2" event={"ID":"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd","Type":"ContainerStarted","Data":"fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52"}
Jul 17 19:22:55.255412 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:55.255010    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:22:55.272186 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:55.272106    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2" podStartSLOduration=1.931618223 podStartE2EDuration="2.272090005s" totalImagesPullingTime="340.471782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:55.271263363 +0000 UTC m=+2984.182090134" watchObservedRunningTime="2026-07-17 19:22:55.272090005 +0000 UTC m=+2984.182916766"
Jul 17 19:22:57.689970 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:57.689922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:22:57.693569 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:57.693538    2563 scope.go:117] "RemoveContainer" containerID="1ddbb25b96fadfd7adabc524c997ff3568d601a4d18ad91e1995927c8d448490"
Jul 17 19:22:58.270092 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:58.270058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" event={"ID":"a9fc4e99-80f8-4c4b-987b-fe8250f98f07","Type":"ContainerStarted","Data":"c1e35cb2ccc0879fb4894dc0e2d50eecb58c2a8201930dd144bb924760d616a7"}
Jul 17 19:22:59.277335 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:59.277297    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerID="c1e35cb2ccc0879fb4894dc0e2d50eecb58c2a8201930dd144bb924760d616a7" exitCode=6
Jul 17 19:22:59.277762 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:59.277367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" event={"ID":"a9fc4e99-80f8-4c4b-987b-fe8250f98f07","Type":"ContainerDied","Data":"c1e35cb2ccc0879fb4894dc0e2d50eecb58c2a8201930dd144bb924760d616a7"}
Jul 17 19:22:59.277762 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:59.277443    2563 scope.go:117] "RemoveContainer" containerID="1ddbb25b96fadfd7adabc524c997ff3568d601a4d18ad91e1995927c8d448490"
Jul 17 19:22:59.277871 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:59.277855    2563 scope.go:117] "RemoveContainer" containerID="c1e35cb2ccc0879fb4894dc0e2d50eecb58c2a8201930dd144bb924760d616a7"
Jul 17 19:22:59.278144 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:22:59.278097    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm_odh-ai-gateway-infra(a9fc4e99-80f8-4c4b-987b-fe8250f98f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07"
Jul 17 19:22:59.302113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:22:59.302075    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"]
Jul 17 19:23:00.234372 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:00.234324    2563 patch_prober.go:28] interesting pod/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 19:23:00.234599 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:00.234387    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 19:23:00.421271 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:00.421241    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"
Jul 17 19:23:00.544359 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:00.544255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc4e99-80f8-4c4b-987b-fe8250f98f07-maas-api-service-ca\") pod \"a9fc4e99-80f8-4c4b-987b-fe8250f98f07\" (UID: \"a9fc4e99-80f8-4c4b-987b-fe8250f98f07\") "
Jul 17 19:23:00.544656 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:00.544633    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fc4e99-80f8-4c4b-987b-fe8250f98f07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9fc4e99-80f8-4c4b-987b-fe8250f98f07" (UID: "a9fc4e99-80f8-4c4b-987b-fe8250f98f07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:23:00.645568 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:00.645523    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc4e99-80f8-4c4b-987b-fe8250f98f07-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:23:01.265018 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.264941    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:23:01.296200 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.296172    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"
Jul 17 19:23:01.296412 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.296220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm" event={"ID":"a9fc4e99-80f8-4c4b-987b-fe8250f98f07","Type":"ContainerDied","Data":"4e1db05fee1a25ad11afd04a4934e3261983f8415eb528775d5a57e8e1fee03f"}
Jul 17 19:23:01.296412 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.296307    2563 scope.go:117] "RemoveContainer" containerID="c1e35cb2ccc0879fb4894dc0e2d50eecb58c2a8201930dd144bb924760d616a7"
Jul 17 19:23:01.298743 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.298718    2563 generic.go:356] "Generic (PLEG): container finished" podID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerID="e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8" exitCode=0
Jul 17 19:23:01.298890 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.298765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" event={"ID":"82ca82f7-e0d9-472f-b3ec-39e74a2afc30","Type":"ContainerDied","Data":"e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8"}
Jul 17 19:23:01.299209 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.299190    2563 scope.go:117] "RemoveContainer" containerID="e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8"
Jul 17 19:23:01.300437 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.300399    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:23:01.300567 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.300499    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:23:01.300567 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.300549    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:23:01.336655 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.336583    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"]
Jul 17 19:23:01.339727 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.339703    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vkdrm"]
Jul 17 19:23:01.695747 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:01.695714    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" path="/var/lib/kubelet/pods/a9fc4e99-80f8-4c4b-987b-fe8250f98f07/volumes"
Jul 17 19:23:02.305947 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:02.305913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" event={"ID":"82ca82f7-e0d9-472f-b3ec-39e74a2afc30","Type":"ContainerStarted","Data":"2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332"}
Jul 17 19:23:02.448695 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:02.448661    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:23:02.448989 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:02.448953    2563 patch_prober.go:28] interesting pod/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 17 19:23:02.449138 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:02.449020    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 17 19:23:03.448902 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:03.448870    2563 patch_prober.go:28] interesting pod/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 17 19:23:03.449336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:03.448925    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 17 19:23:04.448899 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:04.448865    2563 patch_prober.go:28] interesting pod/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 17 19:23:04.449091 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:04.448925    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 17 19:23:05.448933 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:05.448894    2563 patch_prober.go:28] interesting pod/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 17 19:23:05.449145 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:05.448951    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 17 19:23:05.800519 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:05.800420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:23:06.452742 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:06.452712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:23:06.453104 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:06.453032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:23:06.453661 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:06.453641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:23:06.623861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:06.623828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:23:06.726164 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:06.726054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:23:11.731041 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:11.731005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:23:11.736432 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:11.736411    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:23:14.726575 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:14.726546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:23:14.827287 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:14.827252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:23:24.487827 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:24.487780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:23:27.906252 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:27.906218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:23:39.611368 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:39.611323    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:23:45.530333 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.530268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:23:45.954656 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954619    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"]
Jul 17 19:23:45.954933 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954921    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:23:45.954978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954935    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:23:45.954978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954944    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:23:45.954978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954950    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:23:45.954978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954957    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:23:45.954978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954962    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b8d985-8e28-4ead-940b-182ed6054a20" containerName="revoke-keys"
Jul 17 19:23:45.954978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954970    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:23:45.954978 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.954978    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:23:45.955230 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.955034    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:23:45.955230 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.955044    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:23:45.958262 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.958239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"
Jul 17 19:23:45.960714 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.960694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:23:45.967531 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:45.967503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"]
Jul 17 19:23:46.048543 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:46.048499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8538a560-bb2f-4cbd-81a0-f465aebe4670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q\" (UID: \"8538a560-bb2f-4cbd-81a0-f465aebe4670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"
Jul 17 19:23:46.149459 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:46.149423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8538a560-bb2f-4cbd-81a0-f465aebe4670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q\" (UID: \"8538a560-bb2f-4cbd-81a0-f465aebe4670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"
Jul 17 19:23:46.150066 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:46.150042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8538a560-bb2f-4cbd-81a0-f465aebe4670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q\" (UID: \"8538a560-bb2f-4cbd-81a0-f465aebe4670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"
Jul 17 19:23:46.270504 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:46.270427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"
Jul 17 19:23:46.405321 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:46.405292    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"]
Jul 17 19:23:46.408241 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:23:46.408208    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8538a560_bb2f_4cbd_81a0_f465aebe4670.slice/crio-cdfb0b8c55f46eeb4858ada028a071ae412285c0cb63a87fe9f7ae0ddf8fd84a WatchSource:0}: Error finding container cdfb0b8c55f46eeb4858ada028a071ae412285c0cb63a87fe9f7ae0ddf8fd84a: Status 404 returned error can't find the container with id cdfb0b8c55f46eeb4858ada028a071ae412285c0cb63a87fe9f7ae0ddf8fd84a
Jul 17 19:23:46.492835 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:46.492798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q" event={"ID":"8538a560-bb2f-4cbd-81a0-f465aebe4670","Type":"ContainerStarted","Data":"cdfb0b8c55f46eeb4858ada028a071ae412285c0cb63a87fe9f7ae0ddf8fd84a"}
Jul 17 19:23:47.498518 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:47.498486    2563 generic.go:356] "Generic (PLEG): container finished" podID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerID="7add3b7d764cd41f926f9a4e824e7142e5a15eb8f7d0ef885d7f5f35d9c3c8ee" exitCode=6
Jul 17 19:23:47.498917 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:47.498524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q" event={"ID":"8538a560-bb2f-4cbd-81a0-f465aebe4670","Type":"ContainerDied","Data":"7add3b7d764cd41f926f9a4e824e7142e5a15eb8f7d0ef885d7f5f35d9c3c8ee"}
Jul 17 19:23:47.498917 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:47.498715    2563 scope.go:117] "RemoveContainer" containerID="7add3b7d764cd41f926f9a4e824e7142e5a15eb8f7d0ef885d7f5f35d9c3c8ee"
Jul 17 19:23:48.504729 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:48.504642    2563 generic.go:356] "Generic (PLEG): container finished" podID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerID="312e28fed1bdb54e40c8b72e46e9a2bcedc873c1e5e96c62c49f238c3b86d3d6" exitCode=6
Jul 17 19:23:48.504729 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:48.504700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q" event={"ID":"8538a560-bb2f-4cbd-81a0-f465aebe4670","Type":"ContainerDied","Data":"312e28fed1bdb54e40c8b72e46e9a2bcedc873c1e5e96c62c49f238c3b86d3d6"}
Jul 17 19:23:48.505239 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:48.504750    2563 scope.go:117] "RemoveContainer" containerID="7add3b7d764cd41f926f9a4e824e7142e5a15eb8f7d0ef885d7f5f35d9c3c8ee"
Jul 17 19:23:48.505239 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:48.504932    2563 scope.go:117] "RemoveContainer" containerID="312e28fed1bdb54e40c8b72e46e9a2bcedc873c1e5e96c62c49f238c3b86d3d6"
Jul 17 19:23:48.505239 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:23:48.505166    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q_odh-ai-gateway-infra(8538a560-bb2f-4cbd-81a0-f465aebe4670)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670"
Jul 17 19:23:49.510138 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:49.510090    2563 scope.go:117] "RemoveContainer" containerID="312e28fed1bdb54e40c8b72e46e9a2bcedc873c1e5e96c62c49f238c3b86d3d6"
Jul 17 19:23:49.510520 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:23:49.510272    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q_odh-ai-gateway-infra(8538a560-bb2f-4cbd-81a0-f465aebe4670)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670"
Jul 17 19:23:51.202180 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:23:51.202143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:24:01.691100 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:01.691008    2563 scope.go:117] "RemoveContainer" containerID="312e28fed1bdb54e40c8b72e46e9a2bcedc873c1e5e96c62c49f238c3b86d3d6"
Jul 17 19:24:02.563511 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:02.563417    2563 generic.go:356] "Generic (PLEG): container finished" podID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerID="ac44fadb3246b5f22fc0449f984f7b4a90f60854d61267b756aebb505522b925" exitCode=6
Jul 17 19:24:02.563511 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:02.563495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q" event={"ID":"8538a560-bb2f-4cbd-81a0-f465aebe4670","Type":"ContainerDied","Data":"ac44fadb3246b5f22fc0449f984f7b4a90f60854d61267b756aebb505522b925"}
Jul 17 19:24:02.563751 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:02.563571    2563 scope.go:117] "RemoveContainer" containerID="312e28fed1bdb54e40c8b72e46e9a2bcedc873c1e5e96c62c49f238c3b86d3d6"
Jul 17 19:24:02.563921 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:02.563899    2563 scope.go:117] "RemoveContainer" containerID="ac44fadb3246b5f22fc0449f984f7b4a90f60854d61267b756aebb505522b925"
Jul 17 19:24:02.564188 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:02.564168    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q_odh-ai-gateway-infra(8538a560-bb2f-4cbd-81a0-f465aebe4670)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670"
Jul 17 19:24:03.588472 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:03.588429    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"]
Jul 17 19:24:03.724834 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:03.724810    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"
Jul 17 19:24:03.809842 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:03.809810    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8538a560-bb2f-4cbd-81a0-f465aebe4670-maas-api-service-ca\") pod \"8538a560-bb2f-4cbd-81a0-f465aebe4670\" (UID: \"8538a560-bb2f-4cbd-81a0-f465aebe4670\") "
Jul 17 19:24:03.810262 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:03.810239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8538a560-bb2f-4cbd-81a0-f465aebe4670-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8538a560-bb2f-4cbd-81a0-f465aebe4670" (UID: "8538a560-bb2f-4cbd-81a0-f465aebe4670"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:03.911319 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:03.911276    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8538a560-bb2f-4cbd-81a0-f465aebe4670-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:04.573753 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:04.573722    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"
Jul 17 19:24:04.573939 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:04.573722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q" event={"ID":"8538a560-bb2f-4cbd-81a0-f465aebe4670","Type":"ContainerDied","Data":"cdfb0b8c55f46eeb4858ada028a071ae412285c0cb63a87fe9f7ae0ddf8fd84a"}
Jul 17 19:24:04.573939 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:04.573865    2563 scope.go:117] "RemoveContainer" containerID="ac44fadb3246b5f22fc0449f984f7b4a90f60854d61267b756aebb505522b925"
Jul 17 19:24:04.593682 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:04.593652    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"]
Jul 17 19:24:04.598905 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:04.598882    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-x6z7q"]
Jul 17 19:24:05.111243 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:05.111208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:24:05.694629 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:05.694594    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" path="/var/lib/kubelet/pods/8538a560-bb2f-4cbd-81a0-f465aebe4670/volumes"
Jul 17 19:24:11.477336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:11.477307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:12.345504 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.345472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:12.472640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.472601    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"]
Jul 17 19:24:12.472928 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.472915    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:12.472974 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.472933    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:12.472974 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.472960    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:12.472974 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.472966    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:12.473093 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.473021    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:12.473093 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.473031    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:12.473093 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.473038    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9fc4e99-80f8-4c4b-987b-fe8250f98f07" containerName="revoke-keys"
Jul 17 19:24:12.477420 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.477402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"
Jul 17 19:24:12.479963 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.479922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:24:12.482374 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.482341    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"]
Jul 17 19:24:12.589354 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.589306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfd88b1-4aff-4e5f-a4f2-f9398f363270-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s\" (UID: \"8dfd88b1-4aff-4e5f-a4f2-f9398f363270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"
Jul 17 19:24:12.690553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.690470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfd88b1-4aff-4e5f-a4f2-f9398f363270-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s\" (UID: \"8dfd88b1-4aff-4e5f-a4f2-f9398f363270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"
Jul 17 19:24:12.691074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.691054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfd88b1-4aff-4e5f-a4f2-f9398f363270-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s\" (UID: \"8dfd88b1-4aff-4e5f-a4f2-f9398f363270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"
Jul 17 19:24:12.788490 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.788448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"
Jul 17 19:24:12.921503 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:12.921469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"]
Jul 17 19:24:12.924952 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:24:12.924922    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfd88b1_4aff_4e5f_a4f2_f9398f363270.slice/crio-23ac72cf2e730f1ad0b613ad8526fa1bf232d0fda4d2b675f504bb50885d25be WatchSource:0}: Error finding container 23ac72cf2e730f1ad0b613ad8526fa1bf232d0fda4d2b675f504bb50885d25be: Status 404 returned error can't find the container with id 23ac72cf2e730f1ad0b613ad8526fa1bf232d0fda4d2b675f504bb50885d25be
Jul 17 19:24:13.026060 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.026025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:13.042981 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.042949    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"]
Jul 17 19:24:13.043296 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.043282    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:13.043347 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.043298    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:13.043386 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.043354    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8538a560-bb2f-4cbd-81a0-f465aebe4670" containerName="revoke-keys"
Jul 17 19:24:13.047237 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.047221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"
Jul 17 19:24:13.054258 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.054177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"]
Jul 17 19:24:13.094417 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.094387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e779f402-b6a3-4a57-9e10-59a44b404b33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t\" (UID: \"e779f402-b6a3-4a57-9e10-59a44b404b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"
Jul 17 19:24:13.126165 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.126106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:13.195205 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.195178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e779f402-b6a3-4a57-9e10-59a44b404b33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t\" (UID: \"e779f402-b6a3-4a57-9e10-59a44b404b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"
Jul 17 19:24:13.195814 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.195795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e779f402-b6a3-4a57-9e10-59a44b404b33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t\" (UID: \"e779f402-b6a3-4a57-9e10-59a44b404b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"
Jul 17 19:24:13.358474 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.358437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"
Jul 17 19:24:13.490526 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.490502    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"]
Jul 17 19:24:13.493008 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:24:13.492981    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode779f402_b6a3_4a57_9e10_59a44b404b33.slice/crio-5791bbb62a0069b277a8680f41381463817af57a4e51de6a7b153f90ee56d296 WatchSource:0}: Error finding container 5791bbb62a0069b277a8680f41381463817af57a4e51de6a7b153f90ee56d296: Status 404 returned error can't find the container with id 5791bbb62a0069b277a8680f41381463817af57a4e51de6a7b153f90ee56d296
Jul 17 19:24:13.612775 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.612743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t" event={"ID":"e779f402-b6a3-4a57-9e10-59a44b404b33","Type":"ContainerStarted","Data":"5791bbb62a0069b277a8680f41381463817af57a4e51de6a7b153f90ee56d296"}
Jul 17 19:24:13.614229 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.614203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s" event={"ID":"8dfd88b1-4aff-4e5f-a4f2-f9398f363270","Type":"ContainerStarted","Data":"ee5823b5fced67d13602e4abb10b04578bdabcbcae5bb78652c6b95a5298db68"}
Jul 17 19:24:13.614398 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.614243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s" event={"ID":"8dfd88b1-4aff-4e5f-a4f2-f9398f363270","Type":"ContainerStarted","Data":"23ac72cf2e730f1ad0b613ad8526fa1bf232d0fda4d2b675f504bb50885d25be"}
Jul 17 19:24:13.629770 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:13.629712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s" podStartSLOduration=1.629698938 podStartE2EDuration="1.629698938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:24:13.627993726 +0000 UTC m=+3062.538820499" watchObservedRunningTime="2026-07-17 19:24:13.629698938 +0000 UTC m=+3062.540525677"
Jul 17 19:24:14.619394 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:14.619355    2563 generic.go:356] "Generic (PLEG): container finished" podID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerID="e8ca667ead045a01b78302a3cbbc53192d9cd67d0dc017f3394482639e62fa74" exitCode=6
Jul 17 19:24:14.619855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:14.619438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t" event={"ID":"e779f402-b6a3-4a57-9e10-59a44b404b33","Type":"ContainerDied","Data":"e8ca667ead045a01b78302a3cbbc53192d9cd67d0dc017f3394482639e62fa74"}
Jul 17 19:24:14.619855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:14.619600    2563 scope.go:117] "RemoveContainer" containerID="e8ca667ead045a01b78302a3cbbc53192d9cd67d0dc017f3394482639e62fa74"
Jul 17 19:24:14.620933 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:14.620909    2563 generic.go:356] "Generic (PLEG): container finished" podID="8dfd88b1-4aff-4e5f-a4f2-f9398f363270" containerID="ee5823b5fced67d13602e4abb10b04578bdabcbcae5bb78652c6b95a5298db68" exitCode=0
Jul 17 19:24:14.621028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:14.620947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s" event={"ID":"8dfd88b1-4aff-4e5f-a4f2-f9398f363270","Type":"ContainerDied","Data":"ee5823b5fced67d13602e4abb10b04578bdabcbcae5bb78652c6b95a5298db68"}
Jul 17 19:24:15.627315 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:15.627278    2563 generic.go:356] "Generic (PLEG): container finished" podID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerID="7102ff093185f414be72aacdf8337e27b5ec1ae1c6cb2d4fadc0491d4c56289a" exitCode=6
Jul 17 19:24:15.627786 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:15.627368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t" event={"ID":"e779f402-b6a3-4a57-9e10-59a44b404b33","Type":"ContainerDied","Data":"7102ff093185f414be72aacdf8337e27b5ec1ae1c6cb2d4fadc0491d4c56289a"}
Jul 17 19:24:15.627786 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:15.627438    2563 scope.go:117] "RemoveContainer" containerID="e8ca667ead045a01b78302a3cbbc53192d9cd67d0dc017f3394482639e62fa74"
Jul 17 19:24:15.627786 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:15.627658    2563 scope.go:117] "RemoveContainer" containerID="7102ff093185f414be72aacdf8337e27b5ec1ae1c6cb2d4fadc0491d4c56289a"
Jul 17 19:24:15.628323 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:15.627915    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t_odh-ai-gateway-infra(e779f402-b6a3-4a57-9e10-59a44b404b33)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33"
Jul 17 19:24:15.756570 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:15.756544    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"
Jul 17 19:24:15.816135 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:15.816079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfd88b1-4aff-4e5f-a4f2-f9398f363270-maas-api-service-ca\") pod \"8dfd88b1-4aff-4e5f-a4f2-f9398f363270\" (UID: \"8dfd88b1-4aff-4e5f-a4f2-f9398f363270\") "
Jul 17 19:24:15.816517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:15.816494    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfd88b1-4aff-4e5f-a4f2-f9398f363270-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8dfd88b1-4aff-4e5f-a4f2-f9398f363270" (UID: "8dfd88b1-4aff-4e5f-a4f2-f9398f363270"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:15.917554 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:15.917440    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dfd88b1-4aff-4e5f-a4f2-f9398f363270-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:16.633109 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:16.633079    2563 scope.go:117] "RemoveContainer" containerID="7102ff093185f414be72aacdf8337e27b5ec1ae1c6cb2d4fadc0491d4c56289a"
Jul 17 19:24:16.633584 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:16.633312    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t_odh-ai-gateway-infra(e779f402-b6a3-4a57-9e10-59a44b404b33)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33"
Jul 17 19:24:16.634058 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:16.634040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s"
Jul 17 19:24:16.634162 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:16.634082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s" event={"ID":"8dfd88b1-4aff-4e5f-a4f2-f9398f363270","Type":"ContainerDied","Data":"23ac72cf2e730f1ad0b613ad8526fa1bf232d0fda4d2b675f504bb50885d25be"}
Jul 17 19:24:16.634200 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:16.634166    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23ac72cf2e730f1ad0b613ad8526fa1bf232d0fda4d2b675f504bb50885d25be"
Jul 17 19:24:17.905227 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:17.905192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:24:22.899654 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:22.899620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"]
Jul 17 19:24:22.900063 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:22.899826    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk" podUID="c1a29993-4df1-4262-96b3-a998db8587cf" containerName="maas-api" containerID="cri-o://08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb" gracePeriod=30
Jul 17 19:24:22.927680 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:22.927650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:23.146812 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.146787    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:24:23.180602 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.180519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a29993-4df1-4262-96b3-a998db8587cf-openshift-service-ca\") pod \"c1a29993-4df1-4262-96b3-a998db8587cf\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") "
Jul 17 19:24:23.180602 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.180566    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls\") pod \"c1a29993-4df1-4262-96b3-a998db8587cf\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") "
Jul 17 19:24:23.180781 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.180601    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d97x5\" (UniqueName: \"kubernetes.io/projected/c1a29993-4df1-4262-96b3-a998db8587cf-kube-api-access-d97x5\") pod \"c1a29993-4df1-4262-96b3-a998db8587cf\" (UID: \"c1a29993-4df1-4262-96b3-a998db8587cf\") "
Jul 17 19:24:23.180946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.180918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a29993-4df1-4262-96b3-a998db8587cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1a29993-4df1-4262-96b3-a998db8587cf" (UID: "c1a29993-4df1-4262-96b3-a998db8587cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:23.182856 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.182834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1a29993-4df1-4262-96b3-a998db8587cf" (UID: "c1a29993-4df1-4262-96b3-a998db8587cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:24:23.182965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.182947    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a29993-4df1-4262-96b3-a998db8587cf-kube-api-access-d97x5" (OuterVolumeSpecName: "kube-api-access-d97x5") pod "c1a29993-4df1-4262-96b3-a998db8587cf" (UID: "c1a29993-4df1-4262-96b3-a998db8587cf"). InnerVolumeSpecName "kube-api-access-d97x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:24:23.281386 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.281348    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a29993-4df1-4262-96b3-a998db8587cf-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:23.281386 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.281380    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a29993-4df1-4262-96b3-a998db8587cf-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:23.281386 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.281389    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97x5\" (UniqueName: \"kubernetes.io/projected/c1a29993-4df1-4262-96b3-a998db8587cf-kube-api-access-d97x5\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:23.664900 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.664865    2563 generic.go:356] "Generic (PLEG): container finished" podID="c1a29993-4df1-4262-96b3-a998db8587cf" containerID="08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb" exitCode=0
Jul 17 19:24:23.665074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.664914    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"
Jul 17 19:24:23.665074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.664933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk" event={"ID":"c1a29993-4df1-4262-96b3-a998db8587cf","Type":"ContainerDied","Data":"08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb"}
Jul 17 19:24:23.665074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.664975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk" event={"ID":"c1a29993-4df1-4262-96b3-a998db8587cf","Type":"ContainerDied","Data":"718063d5774e775b9b49727cf43104cdc4ecf14f9af098dfb2775aa562ee8385"}
Jul 17 19:24:23.665074 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.664998    2563 scope.go:117] "RemoveContainer" containerID="08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb"
Jul 17 19:24:23.674180 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.674164    2563 scope.go:117] "RemoveContainer" containerID="08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb"
Jul 17 19:24:23.674429 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:23.674412    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb\": container with ID starting with 08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb not found: ID does not exist" containerID="08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb"
Jul 17 19:24:23.674483 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.674437    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb"} err="failed to get container status \"08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb\": rpc error: code = NotFound desc = could not find container \"08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb\": container with ID starting with 08c3300c9068fd5d28e91fb39d62be5fdc638f4c2b719c1ddade22c398fc77fb not found: ID does not exist"
Jul 17 19:24:23.687456 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.687433    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"]
Jul 17 19:24:23.695001 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:23.694980    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4aee7c-b6f9cb7bc-vpxbk"]
Jul 17 19:24:25.694138 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:25.694083    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a29993-4df1-4262-96b3-a998db8587cf" path="/var/lib/kubelet/pods/c1a29993-4df1-4262-96b3-a998db8587cf/volumes"
Jul 17 19:24:27.573699 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:27.573664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:27.689813 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:27.689781    2563 scope.go:117] "RemoveContainer" containerID="7102ff093185f414be72aacdf8337e27b5ec1ae1c6cb2d4fadc0491d4c56289a"
Jul 17 19:24:28.688599 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:28.688565    2563 generic.go:356] "Generic (PLEG): container finished" podID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerID="a02164882088698ac81657fbb331d226c4f42d2841749cc12ae26dbd5cfeaa85" exitCode=6
Jul 17 19:24:28.689030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:28.688630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t" event={"ID":"e779f402-b6a3-4a57-9e10-59a44b404b33","Type":"ContainerDied","Data":"a02164882088698ac81657fbb331d226c4f42d2841749cc12ae26dbd5cfeaa85"}
Jul 17 19:24:28.689030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:28.688689    2563 scope.go:117] "RemoveContainer" containerID="7102ff093185f414be72aacdf8337e27b5ec1ae1c6cb2d4fadc0491d4c56289a"
Jul 17 19:24:28.689030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:28.689000    2563 scope.go:117] "RemoveContainer" containerID="a02164882088698ac81657fbb331d226c4f42d2841749cc12ae26dbd5cfeaa85"
Jul 17 19:24:28.689226 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:28.689199    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t_odh-ai-gateway-infra(e779f402-b6a3-4a57-9e10-59a44b404b33)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33"
Jul 17 19:24:29.601677 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:29.601635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:24:29.723303 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:29.723270    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"]
Jul 17 19:24:29.858414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:29.858348    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"
Jul 17 19:24:29.938486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:29.938454    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e779f402-b6a3-4a57-9e10-59a44b404b33-maas-api-service-ca\") pod \"e779f402-b6a3-4a57-9e10-59a44b404b33\" (UID: \"e779f402-b6a3-4a57-9e10-59a44b404b33\") "
Jul 17 19:24:29.938835 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:29.938815    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e779f402-b6a3-4a57-9e10-59a44b404b33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e779f402-b6a3-4a57-9e10-59a44b404b33" (UID: "e779f402-b6a3-4a57-9e10-59a44b404b33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:30.039749 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:30.039715    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e779f402-b6a3-4a57-9e10-59a44b404b33-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:30.699758 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:30.699725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t" event={"ID":"e779f402-b6a3-4a57-9e10-59a44b404b33","Type":"ContainerDied","Data":"5791bbb62a0069b277a8680f41381463817af57a4e51de6a7b153f90ee56d296"}
Jul 17 19:24:30.699758 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:30.699754    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"
Jul 17 19:24:30.699971 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:30.699782    2563 scope.go:117] "RemoveContainer" containerID="a02164882088698ac81657fbb331d226c4f42d2841749cc12ae26dbd5cfeaa85"
Jul 17 19:24:30.720262 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:30.720230    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"]
Jul 17 19:24:30.724268 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:30.724246    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qcw5t"]
Jul 17 19:24:31.694422 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:31.694391    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" path="/var/lib/kubelet/pods/e779f402-b6a3-4a57-9e10-59a44b404b33/volumes"
Jul 17 19:24:34.566266 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:34.566226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:34.632705 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:34.632645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:38.240379 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:38.240345    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"]
Jul 17 19:24:38.240784 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:38.240639    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy" containerID="cri-o://2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332" gracePeriod=30
Jul 17 19:24:38.250710 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:38.250677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:38.485423 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:38.485388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:39.299222 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:39.299188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:39.398548 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:39.398515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:43.499831 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.499806    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:24:43.664076 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-credential-socket\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664306 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4qrt\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-kube-api-access-q4qrt\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664306 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-data\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664306 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664189    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-certs\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664495 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-socket\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664495 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-podinfo\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664495 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664398    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:43.664495 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-envoy\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664495 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664459    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:43.664740 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istiod-ca-cert\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664740 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664533    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-token\") pod \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\" (UID: \"82ca82f7-e0d9-472f-b3ec-39e74a2afc30\") "
Jul 17 19:24:43.664740 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664631    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:43.664898 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664866    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.664898 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664888    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.665018 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664904    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.665018 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.664885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:43.665403 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.665373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-data" (OuterVolumeSpecName: "istio-data") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:43.666665 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.666636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:43.666665 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.666651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:24:43.666796 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.666676    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-kube-api-access-q4qrt" (OuterVolumeSpecName: "kube-api-access-q4qrt") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "kube-api-access-q4qrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:24:43.666796 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.666720    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-token" (OuterVolumeSpecName: "istio-token") pod "82ca82f7-e0d9-472f-b3ec-39e74a2afc30" (UID: "82ca82f7-e0d9-472f-b3ec-39e74a2afc30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:24:43.756476 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.756431    2563 generic.go:356] "Generic (PLEG): container finished" podID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerID="2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332" exitCode=0
Jul 17 19:24:43.756653 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.756492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"
Jul 17 19:24:43.756653 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.756519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" event={"ID":"82ca82f7-e0d9-472f-b3ec-39e74a2afc30","Type":"ContainerDied","Data":"2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332"}
Jul 17 19:24:43.756653 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.756580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s" event={"ID":"82ca82f7-e0d9-472f-b3ec-39e74a2afc30","Type":"ContainerDied","Data":"ce87625277d15b9d26d4c1760f14a6ccfd7ef804b0a91c099e90ae7dcbeffb77"}
Jul 17 19:24:43.756653 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.756608    2563 scope.go:117] "RemoveContainer" containerID="2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332"
Jul 17 19:24:43.765054 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.765021    2563 scope.go:117] "RemoveContainer" containerID="e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8"
Jul 17 19:24:43.765373 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.765349    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.765442 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.765378    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.765442 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.765391    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.765442 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.765404    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.765442 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.765416    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.765442 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.765430    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qrt\" (UniqueName: \"kubernetes.io/projected/82ca82f7-e0d9-472f-b3ec-39e74a2afc30-kube-api-access-q4qrt\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:43.774731 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.774651    2563 scope.go:117] "RemoveContainer" containerID="2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332"
Jul 17 19:24:43.775235 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:43.775199    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332\": container with ID starting with 2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332 not found: ID does not exist" containerID="2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332"
Jul 17 19:24:43.775337 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.775245    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332"} err="failed to get container status \"2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332\": rpc error: code = NotFound desc = could not find container \"2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332\": container with ID starting with 2991c6435822b4a14764d46984ad01bb0009df793ea96860fd0c864d07d11332 not found: ID does not exist"
Jul 17 19:24:43.775337 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.775268    2563 scope.go:117] "RemoveContainer" containerID="e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8"
Jul 17 19:24:43.775666 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:43.775619    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8\": container with ID starting with e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8 not found: ID does not exist" containerID="e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8"
Jul 17 19:24:43.775666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.775651    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8"} err="failed to get container status \"e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8\": rpc error: code = NotFound desc = could not find container \"e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8\": container with ID starting with e4d7ce4ff665c3691c050d475946a04398e29ca6f6ae1f0883bf500623e75ff8 not found: ID does not exist"
Jul 17 19:24:43.776460 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.776433    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"]
Jul 17 19:24:43.782667 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:43.782643    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4aee7c-openshift-default-5678877979-56v6s"]
Jul 17 19:24:44.196727 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.196699    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"]
Jul 17 19:24:44.196980 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.196968    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.196981    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.196990    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.196995    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197006    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfd88b1-4aff-4e5f-a4f2-f9398f363270" containerName="revoke-keys"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197011    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfd88b1-4aff-4e5f-a4f2-f9398f363270" containerName="revoke-keys"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197017    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a29993-4df1-4262-96b3-a998db8587cf" containerName="maas-api"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197022    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a29993-4df1-4262-96b3-a998db8587cf" containerName="maas-api"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197031    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:24:44.197035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197036    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:24:44.197361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197079    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dfd88b1-4aff-4e5f-a4f2-f9398f363270" containerName="revoke-keys"
Jul 17 19:24:44.197361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197088    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a29993-4df1-4262-96b3-a998db8587cf" containerName="maas-api"
Jul 17 19:24:44.197361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197094    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy"
Jul 17 19:24:44.197361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197100    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy"
Jul 17 19:24:44.197361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197106    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:24:44.197361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197111    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:24:44.197361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.197153    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:24:44.201610 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.201592    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"
Jul 17 19:24:44.204287 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.204267    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:24:44.207464 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.207435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"]
Jul 17 19:24:44.270391 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.270351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ef841-425c-440d-9a30-1aa31f83283c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f\" (UID: \"620ef841-425c-440d-9a30-1aa31f83283c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"
Jul 17 19:24:44.371317 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.371274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ef841-425c-440d-9a30-1aa31f83283c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f\" (UID: \"620ef841-425c-440d-9a30-1aa31f83283c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"
Jul 17 19:24:44.371932 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.371910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ef841-425c-440d-9a30-1aa31f83283c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f\" (UID: \"620ef841-425c-440d-9a30-1aa31f83283c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"
Jul 17 19:24:44.513044 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.512948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"
Jul 17 19:24:44.644330 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.644296    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"]
Jul 17 19:24:44.647342 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:24:44.647309    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620ef841_425c_440d_9a30_1aa31f83283c.slice/crio-0a37ef268c093e1c575f566075468cf9011294159ee63983af9709e1118b18e5 WatchSource:0}: Error finding container 0a37ef268c093e1c575f566075468cf9011294159ee63983af9709e1118b18e5: Status 404 returned error can't find the container with id 0a37ef268c093e1c575f566075468cf9011294159ee63983af9709e1118b18e5
Jul 17 19:24:44.762889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:44.762853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f" event={"ID":"620ef841-425c-440d-9a30-1aa31f83283c","Type":"ContainerStarted","Data":"0a37ef268c093e1c575f566075468cf9011294159ee63983af9709e1118b18e5"}
Jul 17 19:24:45.694317 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:45.694284    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" path="/var/lib/kubelet/pods/82ca82f7-e0d9-472f-b3ec-39e74a2afc30/volumes"
Jul 17 19:24:45.768543 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:45.768511    2563 generic.go:356] "Generic (PLEG): container finished" podID="620ef841-425c-440d-9a30-1aa31f83283c" containerID="8d49c357f5a6e715a56629b7cede971406b1c4862a736b2600373008a9960317" exitCode=6
Jul 17 19:24:45.768728 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:45.768579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f" event={"ID":"620ef841-425c-440d-9a30-1aa31f83283c","Type":"ContainerDied","Data":"8d49c357f5a6e715a56629b7cede971406b1c4862a736b2600373008a9960317"}
Jul 17 19:24:45.768777 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:45.768764    2563 scope.go:117] "RemoveContainer" containerID="8d49c357f5a6e715a56629b7cede971406b1c4862a736b2600373008a9960317"
Jul 17 19:24:46.775556 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:46.775457    2563 generic.go:356] "Generic (PLEG): container finished" podID="620ef841-425c-440d-9a30-1aa31f83283c" containerID="7a0dbadae05a053b84d8229b6e1863090b097c9e8251d9b7c199a671a66a71d7" exitCode=6
Jul 17 19:24:46.775556 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:46.775496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f" event={"ID":"620ef841-425c-440d-9a30-1aa31f83283c","Type":"ContainerDied","Data":"7a0dbadae05a053b84d8229b6e1863090b097c9e8251d9b7c199a671a66a71d7"}
Jul 17 19:24:46.775998 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:46.775561    2563 scope.go:117] "RemoveContainer" containerID="8d49c357f5a6e715a56629b7cede971406b1c4862a736b2600373008a9960317"
Jul 17 19:24:46.775998 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:46.775764    2563 scope.go:117] "RemoveContainer" containerID="7a0dbadae05a053b84d8229b6e1863090b097c9e8251d9b7c199a671a66a71d7"
Jul 17 19:24:46.775998 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:46.775988    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f_odh-ai-gateway-infra(620ef841-425c-440d-9a30-1aa31f83283c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f" podUID="620ef841-425c-440d-9a30-1aa31f83283c"
Jul 17 19:24:47.781243 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:47.781219    2563 scope.go:117] "RemoveContainer" containerID="7a0dbadae05a053b84d8229b6e1863090b097c9e8251d9b7c199a671a66a71d7"
Jul 17 19:24:47.781624 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:47.781395    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f_odh-ai-gateway-infra(620ef841-425c-440d-9a30-1aa31f83283c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f" podUID="620ef841-425c-440d-9a30-1aa31f83283c"
Jul 17 19:24:49.011533 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.011431    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"]
Jul 17 19:24:49.012016 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.011691    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2" podUID="46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" containerName="maas-api" containerID="cri-o://fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52" gracePeriod=30
Jul 17 19:24:49.037155 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.037110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:49.267546 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.267481    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:24:49.416624 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.416575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-maas-api-tls\") pod \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") "
Jul 17 19:24:49.416803 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.416662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxbqj\" (UniqueName: \"kubernetes.io/projected/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-kube-api-access-zxbqj\") pod \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") "
Jul 17 19:24:49.416803 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.416759    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-openshift-service-ca\") pod \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\" (UID: \"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd\") "
Jul 17 19:24:49.417140 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.417091    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" (UID: "46a92362-0a05-4a5a-91c4-f3a2ad6be4fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:49.418945 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.418924    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" (UID: "46a92362-0a05-4a5a-91c4-f3a2ad6be4fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:24:49.419021 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.418971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-kube-api-access-zxbqj" (OuterVolumeSpecName: "kube-api-access-zxbqj") pod "46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" (UID: "46a92362-0a05-4a5a-91c4-f3a2ad6be4fd"). InnerVolumeSpecName "kube-api-access-zxbqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:24:49.517640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.517539    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxbqj\" (UniqueName: \"kubernetes.io/projected/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-kube-api-access-zxbqj\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:49.517640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.517581    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:49.517640 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.517591    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:24:49.789717 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.789617    2563 generic.go:356] "Generic (PLEG): container finished" podID="46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" containerID="fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52" exitCode=0
Jul 17 19:24:49.789717 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.789667    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"
Jul 17 19:24:49.789717 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.789684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2" event={"ID":"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd","Type":"ContainerDied","Data":"fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52"}
Jul 17 19:24:49.789986 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.789735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2" event={"ID":"46a92362-0a05-4a5a-91c4-f3a2ad6be4fd","Type":"ContainerDied","Data":"3c41e07e854563bc600cbbacc0fdfa4c05e52c58803b814990c96ceea52d5fc5"}
Jul 17 19:24:49.789986 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.789768    2563 scope.go:117] "RemoveContainer" containerID="fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52"
Jul 17 19:24:49.798090 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.798065    2563 scope.go:117] "RemoveContainer" containerID="fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52"
Jul 17 19:24:49.798490 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:24:49.798471    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52\": container with ID starting with fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52 not found: ID does not exist" containerID="fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52"
Jul 17 19:24:49.798560 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.798499    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52"} err="failed to get container status \"fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52\": rpc error: code = NotFound desc = could not find container \"fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52\": container with ID starting with fed0780707c0cee81b4153dba883a9938e93823ef428be51664f19b83e436d52 not found: ID does not exist"
Jul 17 19:24:49.806738 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.806711    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"]
Jul 17 19:24:49.808073 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:49.808053    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15f29d-787b9cd456-fhzq2"]
Jul 17 19:24:51.694586 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:51.694543    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" path="/var/lib/kubelet/pods/46a92362-0a05-4a5a-91c4-f3a2ad6be4fd/volumes"
Jul 17 19:24:52.105941 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:52.105899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:24:53.684384 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:53.684347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:24:59.688903 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:59.688863    2563 scope.go:117] "RemoveContainer" containerID="7a0dbadae05a053b84d8229b6e1863090b097c9e8251d9b7c199a671a66a71d7"
Jul 17 19:24:59.900234 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:24:59.900147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:00.001031 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:00.000922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:00.839538 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:00.839505    2563 generic.go:356] "Generic (PLEG): container finished" podID="620ef841-425c-440d-9a30-1aa31f83283c" containerID="c1b13a4281b28aff0313f3b9fcdf7104940dbc54b3c82d7a7156aa2d5c64e28e" exitCode=6
Jul 17 19:25:00.839949 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:00.839575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f" event={"ID":"620ef841-425c-440d-9a30-1aa31f83283c","Type":"ContainerDied","Data":"c1b13a4281b28aff0313f3b9fcdf7104940dbc54b3c82d7a7156aa2d5c64e28e"}
Jul 17 19:25:00.839949 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:00.839635    2563 scope.go:117] "RemoveContainer" containerID="7a0dbadae05a053b84d8229b6e1863090b097c9e8251d9b7c199a671a66a71d7"
Jul 17 19:25:00.839949 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:00.839913    2563 scope.go:117] "RemoveContainer" containerID="c1b13a4281b28aff0313f3b9fcdf7104940dbc54b3c82d7a7156aa2d5c64e28e"
Jul 17 19:25:00.840146 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:25:00.840105    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f_odh-ai-gateway-infra(620ef841-425c-440d-9a30-1aa31f83283c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f" podUID="620ef841-425c-440d-9a30-1aa31f83283c"
Jul 17 19:25:01.863693 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:01.863651    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"]
Jul 17 19:25:01.991411 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:01.991390    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"
Jul 17 19:25:02.022256 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:02.022224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ef841-425c-440d-9a30-1aa31f83283c-maas-api-service-ca\") pod \"620ef841-425c-440d-9a30-1aa31f83283c\" (UID: \"620ef841-425c-440d-9a30-1aa31f83283c\") "
Jul 17 19:25:02.022565 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:02.022544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620ef841-425c-440d-9a30-1aa31f83283c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "620ef841-425c-440d-9a30-1aa31f83283c" (UID: "620ef841-425c-440d-9a30-1aa31f83283c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:02.122904 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:02.122795    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620ef841-425c-440d-9a30-1aa31f83283c-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:02.851237 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:02.851206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"
Jul 17 19:25:02.851423 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:02.851207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f" event={"ID":"620ef841-425c-440d-9a30-1aa31f83283c","Type":"ContainerDied","Data":"0a37ef268c093e1c575f566075468cf9011294159ee63983af9709e1118b18e5"}
Jul 17 19:25:02.851423 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:02.851342    2563 scope.go:117] "RemoveContainer" containerID="c1b13a4281b28aff0313f3b9fcdf7104940dbc54b3c82d7a7156aa2d5c64e28e"
Jul 17 19:25:02.871583 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:02.871555    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"]
Jul 17 19:25:02.877112 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:02.877084    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vm54f"]
Jul 17 19:25:03.086225 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086187    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"]
Jul 17 19:25:03.086500 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086487    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:03.086557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086502    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:03.086557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086513    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy"
Jul 17 19:25:03.086557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086519    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ca82f7-e0d9-472f-b3ec-39e74a2afc30" containerName="istio-proxy"
Jul 17 19:25:03.086557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" containerName="maas-api"
Jul 17 19:25:03.086557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086547    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" containerName="maas-api"
Jul 17 19:25:03.086557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086555    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:25:03.086557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e779f402-b6a3-4a57-9e10-59a44b404b33" containerName="revoke-keys"
Jul 17 19:25:03.086767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086571    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:03.086767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086577    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:03.086767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086640    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:03.086767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086652    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:03.086767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.086658    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="46a92362-0a05-4a5a-91c4-f3a2ad6be4fd" containerName="maas-api"
Jul 17 19:25:03.090681 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.090662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"
Jul 17 19:25:03.093217 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.093197    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:25:03.093217 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.093213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:25:03.105082 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.100377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"]
Jul 17 19:25:03.130216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.130170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f38c99-9852-4bee-a01f-16cd289868dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj\" (UID: \"90f38c99-9852-4bee-a01f-16cd289868dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"
Jul 17 19:25:03.230869 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.230831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f38c99-9852-4bee-a01f-16cd289868dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj\" (UID: \"90f38c99-9852-4bee-a01f-16cd289868dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"
Jul 17 19:25:03.231583 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.231561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f38c99-9852-4bee-a01f-16cd289868dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj\" (UID: \"90f38c99-9852-4bee-a01f-16cd289868dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"
Jul 17 19:25:03.403752 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.403653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"
Jul 17 19:25:03.532152 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.532106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"]
Jul 17 19:25:03.534565 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:25:03.534535    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f38c99_9852_4bee_a01f_16cd289868dd.slice/crio-a03d057cbe294fd1b65c4b3c3246e7c1f05bdfc93f758623c26812624f2fd75f WatchSource:0}: Error finding container a03d057cbe294fd1b65c4b3c3246e7c1f05bdfc93f758623c26812624f2fd75f: Status 404 returned error can't find the container with id a03d057cbe294fd1b65c4b3c3246e7c1f05bdfc93f758623c26812624f2fd75f
Jul 17 19:25:03.693253 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.693168    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="620ef841-425c-440d-9a30-1aa31f83283c" path="/var/lib/kubelet/pods/620ef841-425c-440d-9a30-1aa31f83283c/volumes"
Jul 17 19:25:03.856912 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:03.856876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" event={"ID":"90f38c99-9852-4bee-a01f-16cd289868dd","Type":"ContainerStarted","Data":"a03d057cbe294fd1b65c4b3c3246e7c1f05bdfc93f758623c26812624f2fd75f"}
Jul 17 19:25:04.372288 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:04.372251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:04.606476 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:04.606440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:04.863684 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:04.863602    2563 generic.go:356] "Generic (PLEG): container finished" podID="90f38c99-9852-4bee-a01f-16cd289868dd" containerID="358e0edc17a876291968bd6fa5aa89f89d06f01733992dae2eb330db830f69ae" exitCode=6
Jul 17 19:25:04.863842 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:04.863692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" event={"ID":"90f38c99-9852-4bee-a01f-16cd289868dd","Type":"ContainerDied","Data":"358e0edc17a876291968bd6fa5aa89f89d06f01733992dae2eb330db830f69ae"}
Jul 17 19:25:04.863842 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:04.863828    2563 scope.go:117] "RemoveContainer" containerID="358e0edc17a876291968bd6fa5aa89f89d06f01733992dae2eb330db830f69ae"
Jul 17 19:25:04.922482 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:04.922446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:05.215629 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.215549    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"]
Jul 17 19:25:05.215850 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.215838    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:05.215896 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.215851    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:05.215929 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.215897    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="620ef841-425c-440d-9a30-1aa31f83283c" containerName="revoke-keys"
Jul 17 19:25:05.218771 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.218755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.221759 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.221737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-40565f-openshift-default-dockercfg-wjqn4\""
Jul 17 19:25:05.230909 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.230878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"]
Jul 17 19:25:05.243176 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-credential-socket\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.243333 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-podinfo\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.243333 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099b2c46-c022-44a7-a208-c85b4f7f639a-istiod-ca-cert\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.243333 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-envoy\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.243441 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csp8z\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-kube-api-access-csp8z\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.243441 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-socket\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.243441 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-certs\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.243553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-token\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.243553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.243489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-data\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.248146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.248099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:05.344373 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-token\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344594 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-data\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344594 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-credential-socket\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344594 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-podinfo\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344775 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099b2c46-c022-44a7-a208-c85b4f7f639a-istiod-ca-cert\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344775 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-envoy\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344775 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csp8z\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-kube-api-access-csp8z\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344775 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-socket\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344987 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-certs\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344987 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-credential-socket\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.344987 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.344953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-data\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.345399 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.345292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-certs\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.345587 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.345566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-socket\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.345717 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.345690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099b2c46-c022-44a7-a208-c85b4f7f639a-istiod-ca-cert\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.347097 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.347072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-envoy\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.347267 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.347210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-podinfo\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.352494 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.352469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-token\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.352600 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.352572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csp8z\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-kube-api-access-csp8z\") pod \"e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") " pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.529091 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.529054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:05.664044 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.664012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"]
Jul 17 19:25:05.667537 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:25:05.667507    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099b2c46_c022_44a7_a208_c85b4f7f639a.slice/crio-1f7c3097b74188ee2ea465674ef93ea84778ff4414a31bb7c8b8c71c4f42fe89 WatchSource:0}: Error finding container 1f7c3097b74188ee2ea465674ef93ea84778ff4414a31bb7c8b8c71c4f42fe89: Status 404 returned error can't find the container with id 1f7c3097b74188ee2ea465674ef93ea84778ff4414a31bb7c8b8c71c4f42fe89
Jul 17 19:25:05.669869 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.669827    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:05.669969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.669910    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:05.669969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.669953    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:05.780475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.780390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:05.871094 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.871051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" event={"ID":"099b2c46-c022-44a7-a208-c85b4f7f639a","Type":"ContainerStarted","Data":"4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7"}
Jul 17 19:25:05.871332 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.871138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" event={"ID":"099b2c46-c022-44a7-a208-c85b4f7f639a","Type":"ContainerStarted","Data":"1f7c3097b74188ee2ea465674ef93ea84778ff4414a31bb7c8b8c71c4f42fe89"}
Jul 17 19:25:05.873029 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.873003    2563 generic.go:356] "Generic (PLEG): container finished" podID="90f38c99-9852-4bee-a01f-16cd289868dd" containerID="b0e5796dee72d0c30cccf7dea8ad45104ac28e096c1d4d435e6ad8cae7885ebd" exitCode=6
Jul 17 19:25:05.873204 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.873082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" event={"ID":"90f38c99-9852-4bee-a01f-16cd289868dd","Type":"ContainerDied","Data":"b0e5796dee72d0c30cccf7dea8ad45104ac28e096c1d4d435e6ad8cae7885ebd"}
Jul 17 19:25:05.873204 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.873142    2563 scope.go:117] "RemoveContainer" containerID="358e0edc17a876291968bd6fa5aa89f89d06f01733992dae2eb330db830f69ae"
Jul 17 19:25:05.873440 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.873424    2563 scope.go:117] "RemoveContainer" containerID="b0e5796dee72d0c30cccf7dea8ad45104ac28e096c1d4d435e6ad8cae7885ebd"
Jul 17 19:25:05.873855 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:25:05.873630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj_odh-ai-gateway-infra(90f38c99-9852-4bee-a01f-16cd289868dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" podUID="90f38c99-9852-4bee-a01f-16cd289868dd"
Jul 17 19:25:05.891883 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:05.891517    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" podStartSLOduration=0.89149695 podStartE2EDuration="891.49695ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:05.890995797 +0000 UTC m=+3114.801822595" watchObservedRunningTime="2026-07-17 19:25:05.89149695 +0000 UTC m=+3114.802323712"
Jul 17 19:25:06.529502 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:06.529471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:06.534318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:06.534288    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:06.879000 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:06.878972    2563 scope.go:117] "RemoveContainer" containerID="b0e5796dee72d0c30cccf7dea8ad45104ac28e096c1d4d435e6ad8cae7885ebd"
Jul 17 19:25:06.879190 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:06.879024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:06.879257 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:25:06.879227    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj_odh-ai-gateway-infra(90f38c99-9852-4bee-a01f-16cd289868dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" podUID="90f38c99-9852-4bee-a01f-16cd289868dd"
Jul 17 19:25:06.880019 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:06.880003    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:07.808363 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.808320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:07.885861 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.885819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:07.891459 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.891426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"]
Jul 17 19:25:07.896335 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.896311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:07.899041 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.899018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:25:07.899218 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.899026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-40565f-serving-cert\""
Jul 17 19:25:07.899900 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.899885    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:25:07.906707 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.906680    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"]
Jul 17 19:25:07.967384 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.967345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:07.968435 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.968412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-maas-api-tls\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:07.968576 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.968556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-openshift-service-ca\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:07.968687 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:07.968671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zrz\" (UniqueName: \"kubernetes.io/projected/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-kube-api-access-z2zrz\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:08.058188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.058153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:08.069906 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.069819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-openshift-service-ca\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:08.070070 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.070020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zrz\" (UniqueName: \"kubernetes.io/projected/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-kube-api-access-z2zrz\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:08.070158 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.070064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-maas-api-tls\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:08.070687 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.070648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-openshift-service-ca\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:08.078903 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.078876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-maas-api-tls\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:08.081621 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.081593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zrz\" (UniqueName: \"kubernetes.io/projected/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-kube-api-access-z2zrz\") pod \"maas-api-e2e-api-a-40565f-98696c689-kktgq\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:08.208351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.208311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:08.360414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.360387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"]
Jul 17 19:25:08.362646 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:25:08.362609    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f081c1_72bf_4c60_a3f3_a6b0ee280563.slice/crio-06c56df47bb6ecf7ba5f0e683db9f0ee366c7b71cf128fe3bb9dd3628042a435 WatchSource:0}: Error finding container 06c56df47bb6ecf7ba5f0e683db9f0ee366c7b71cf128fe3bb9dd3628042a435: Status 404 returned error can't find the container with id 06c56df47bb6ecf7ba5f0e683db9f0ee366c7b71cf128fe3bb9dd3628042a435
Jul 17 19:25:08.364082 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.364065    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:25:08.499500 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.499465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:08.598641 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.598613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:08.890606 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.890568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq" event={"ID":"f6f081c1-72bf-4c60-a3f3-a6b0ee280563","Type":"ContainerStarted","Data":"83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10"}
Jul 17 19:25:08.890988 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.890635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq" event={"ID":"f6f081c1-72bf-4c60-a3f3-a6b0ee280563","Type":"ContainerStarted","Data":"06c56df47bb6ecf7ba5f0e683db9f0ee366c7b71cf128fe3bb9dd3628042a435"}
Jul 17 19:25:08.913757 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:08.913704    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq" podStartSLOduration=1.471752965 podStartE2EDuration="1.913687801s" totalImagesPullingTime="441.934836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:08.91128646 +0000 UTC m=+3117.822113243" watchObservedRunningTime="2026-07-17 19:25:08.913687801 +0000 UTC m=+3117.824514596"
Jul 17 19:25:09.894317 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:09.894292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:12.361419 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:12.361388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:12.802350 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:12.802264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:25:12.831357 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:12.831325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:13.109403 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.109365    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"]
Jul 17 19:25:13.114152 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.114106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.119589 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.119562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-460276-openshift-default-dockercfg-mw52h\""
Jul 17 19:25:13.131336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.131305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"]
Jul 17 19:25:13.133693 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.133669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:13.214972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.214924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-data\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.215347 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.215307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-credential-socket\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.215482 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.215368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-podinfo\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.215482 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.215410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-token\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.215613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.215535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-certs\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.215720 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.215686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-socket\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.215829 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.215768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-envoy\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.215904 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.215805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7f8\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-kube-api-access-nr7f8\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.215904 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.215877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istiod-ca-cert\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.266109 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.266067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:13.317449 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-certs\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-socket\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-envoy\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7f8\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-kube-api-access-nr7f8\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istiod-ca-cert\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317843 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-data\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317843 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-credential-socket\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317843 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-podinfo\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317993 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-certs\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317993 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-token\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.317993 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.317941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-socket\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.318226 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.318202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-data\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.318303 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.318274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-credential-socket\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.318402 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.318382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istiod-ca-cert\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.320243 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.320225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-envoy\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.320322 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.320268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-podinfo\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.325992 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.325964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-token\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.326313 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.326284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7f8\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-kube-api-access-nr7f8\") pod \"e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") " pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.425235 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.425148    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:13.562948 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.562922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"]
Jul 17 19:25:13.565268 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:25:13.565238    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b27a5a2_a997_422d_8e9a_6fc05c71bb6a.slice/crio-ad57eb896789ab64b41c185f5df8550015d0942a489e7992db75c1926c604309 WatchSource:0}: Error finding container ad57eb896789ab64b41c185f5df8550015d0942a489e7992db75c1926c604309: Status 404 returned error can't find the container with id ad57eb896789ab64b41c185f5df8550015d0942a489e7992db75c1926c604309
Jul 17 19:25:13.567510 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.567476    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:13.567588 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.567546    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:13.567588 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.567575    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:13.911960 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.911925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" event={"ID":"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a","Type":"ContainerStarted","Data":"49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561"}
Jul 17 19:25:13.912188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.911974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" event={"ID":"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a","Type":"ContainerStarted","Data":"ad57eb896789ab64b41c185f5df8550015d0942a489e7992db75c1926c604309"}
Jul 17 19:25:13.932312 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:13.932258    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" podStartSLOduration=0.932241578 podStartE2EDuration="932.241578ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:13.931003286 +0000 UTC m=+3122.841830047" watchObservedRunningTime="2026-07-17 19:25:13.932241578 +0000 UTC m=+3122.843068317"
Jul 17 19:25:14.425865 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:14.425819    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:14.431679 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:14.431655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:14.916132 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:14.916080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:14.917218 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:14.917200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:15.831112 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.831077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:15.905474 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.905443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:15.909720 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.909688    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"]
Jul 17 19:25:15.913717 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.913691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:15.916496 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.916340    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-460276-serving-cert\""
Jul 17 19:25:15.921406 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.921378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:15.928463 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.927879    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"]
Jul 17 19:25:15.940669 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.940630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-maas-api-tls\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:15.940830 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.940676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:15.940830 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.940801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fngv\" (UniqueName: \"kubernetes.io/projected/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-kube-api-access-5fngv\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:15.998972 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:15.998930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:16.041332 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.041279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fngv\" (UniqueName: \"kubernetes.io/projected/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-kube-api-access-5fngv\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:16.041600 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.041399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-maas-api-tls\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:16.041600 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.041549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:16.042240 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.042169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:16.043971 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.043949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-maas-api-tls\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:16.051550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.051503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fngv\" (UniqueName: \"kubernetes.io/projected/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-kube-api-access-5fngv\") pod \"maas-api-e2e-api-b-460276-c66895bf5-88h2z\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:16.097788 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.097748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:16.230388 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.230344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:16.398307 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.398209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"]
Jul 17 19:25:16.403679 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:25:16.403641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211fc4f7_fe0c_4fc0_97c0_3b15bf67bcb1.slice/crio-63a52014553fcc17d2e66349a8d5f7e64bc0e4b570ebec1a84ded90422858f02 WatchSource:0}: Error finding container 63a52014553fcc17d2e66349a8d5f7e64bc0e4b570ebec1a84ded90422858f02: Status 404 returned error can't find the container with id 63a52014553fcc17d2e66349a8d5f7e64bc0e4b570ebec1a84ded90422858f02
Jul 17 19:25:16.634134 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.634086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:16.752157 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.752052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:16.930375 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.930336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z" event={"ID":"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1","Type":"ContainerStarted","Data":"5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26"}
Jul 17 19:25:16.930860 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.930400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:16.930860 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.930421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z" event={"ID":"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1","Type":"ContainerStarted","Data":"63a52014553fcc17d2e66349a8d5f7e64bc0e4b570ebec1a84ded90422858f02"}
Jul 17 19:25:16.952174 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:16.952069    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z" podStartSLOduration=1.557884797 podStartE2EDuration="1.952051937s" totalImagesPullingTime="394.16714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:16.950890244 +0000 UTC m=+3125.861717014" watchObservedRunningTime="2026-07-17 19:25:16.952051937 +0000 UTC m=+3125.862878691"
Jul 17 19:25:17.692818 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:17.692775    2563 scope.go:117] "RemoveContainer" containerID="b0e5796dee72d0c30cccf7dea8ad45104ac28e096c1d4d435e6ad8cae7885ebd"
Jul 17 19:25:17.937806 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:17.937773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" event={"ID":"90f38c99-9852-4bee-a01f-16cd289868dd","Type":"ContainerStarted","Data":"b87925495d1cb2a90b136a2928ec3a29a5e2c2525eb474a5e6ae3b3ea2d9d4a4"}
Jul 17 19:25:17.954015 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:17.953921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" podStartSLOduration=14.953904371 podStartE2EDuration="14.953904371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:17.952714474 +0000 UTC m=+3126.863541246" watchObservedRunningTime="2026-07-17 19:25:17.953904371 +0000 UTC m=+3126.864731133"
Jul 17 19:25:18.943881 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:18.943850    2563 generic.go:356] "Generic (PLEG): container finished" podID="90f38c99-9852-4bee-a01f-16cd289868dd" containerID="b87925495d1cb2a90b136a2928ec3a29a5e2c2525eb474a5e6ae3b3ea2d9d4a4" exitCode=6
Jul 17 19:25:18.944343 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:18.943931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" event={"ID":"90f38c99-9852-4bee-a01f-16cd289868dd","Type":"ContainerDied","Data":"b87925495d1cb2a90b136a2928ec3a29a5e2c2525eb474a5e6ae3b3ea2d9d4a4"}
Jul 17 19:25:18.944343 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:18.944014    2563 scope.go:117] "RemoveContainer" containerID="b0e5796dee72d0c30cccf7dea8ad45104ac28e096c1d4d435e6ad8cae7885ebd"
Jul 17 19:25:18.944343 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:18.944325    2563 scope.go:117] "RemoveContainer" containerID="b87925495d1cb2a90b136a2928ec3a29a5e2c2525eb474a5e6ae3b3ea2d9d4a4"
Jul 17 19:25:18.944617 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:25:18.944597    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj_odh-ai-gateway-infra(90f38c99-9852-4bee-a01f-16cd289868dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" podUID="90f38c99-9852-4bee-a01f-16cd289868dd"
Jul 17 19:25:18.969651 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:18.969611    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"]
Jul 17 19:25:20.085594 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.085570    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"
Jul 17 19:25:20.179635 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.179597    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f38c99-9852-4bee-a01f-16cd289868dd-maas-api-service-ca\") pod \"90f38c99-9852-4bee-a01f-16cd289868dd\" (UID: \"90f38c99-9852-4bee-a01f-16cd289868dd\") "
Jul 17 19:25:20.179969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.179946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f38c99-9852-4bee-a01f-16cd289868dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90f38c99-9852-4bee-a01f-16cd289868dd" (UID: "90f38c99-9852-4bee-a01f-16cd289868dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:20.281097 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.281012    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f38c99-9852-4bee-a01f-16cd289868dd-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:20.329903 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.329870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:20.956186 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.956153    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"
Jul 17 19:25:20.956441 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.956178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj" event={"ID":"90f38c99-9852-4bee-a01f-16cd289868dd","Type":"ContainerDied","Data":"a03d057cbe294fd1b65c4b3c3246e7c1f05bdfc93f758623c26812624f2fd75f"}
Jul 17 19:25:20.956441 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.956245    2563 scope.go:117] "RemoveContainer" containerID="b87925495d1cb2a90b136a2928ec3a29a5e2c2525eb474a5e6ae3b3ea2d9d4a4"
Jul 17 19:25:20.957863 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.957830    2563 generic.go:356] "Generic (PLEG): container finished" podID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerID="4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7" exitCode=0
Jul 17 19:25:20.957970 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.957868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" event={"ID":"099b2c46-c022-44a7-a208-c85b4f7f639a","Type":"ContainerDied","Data":"4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7"}
Jul 17 19:25:20.958276 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.958259    2563 scope.go:117] "RemoveContainer" containerID="4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7"
Jul 17 19:25:20.959357 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.959323    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:20.959447 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.959409    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:20.959498 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.959452    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:20.999957 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:20.999919    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"]
Jul 17 19:25:21.003723 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:21.003696    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-rkwzj"]
Jul 17 19:25:21.697493 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:21.697444    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" path="/var/lib/kubelet/pods/90f38c99-9852-4bee-a01f-16cd289868dd/volumes"
Jul 17 19:25:21.879264 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:21.879204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:21.964694 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:21.964606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" event={"ID":"099b2c46-c022-44a7-a208-c85b4f7f639a","Type":"ContainerStarted","Data":"4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c"}
Jul 17 19:25:22.529439 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:22.529396    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:22.530648 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:22.530622    2563 patch_prober.go:28] interesting pod/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 17 19:25:22.530783 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:22.530671    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 17 19:25:22.943047 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:22.943017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:25:23.530416 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:23.530385    2563 patch_prober.go:28] interesting pod/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 17 19:25:23.530591 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:23.530441    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 17 19:25:23.909814 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:23.909781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:25:24.280761 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:24.280656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:24.360064 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:24.360029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:24.530336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:24.530301    2563 patch_prober.go:28] interesting pod/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 17 19:25:24.530500 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:24.530366    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 17 19:25:25.530165 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:25.530102    2563 patch_prober.go:28] interesting pod/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 17 19:25:25.530568 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:25.530188    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 17 19:25:26.533882 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:26.533849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:26.534275 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:26.534159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:26.534915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:26.534895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:28.903170 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:28.903130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:29.003783 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:29.003748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:30.957314 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:30.957272    2563 patch_prober.go:28] interesting pod/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 19:25:30.957708 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:30.957363    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 19:25:32.017753 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:32.017646    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerID="49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561" exitCode=0
Jul 17 19:25:32.017753 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:32.017703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" event={"ID":"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a","Type":"ContainerDied","Data":"49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561"}
Jul 17 19:25:32.018314 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:32.018101    2563 scope.go:117] "RemoveContainer" containerID="49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561"
Jul 17 19:25:32.019078 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:32.019044    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:32.019203 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:32.019138    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:32.019203 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:32.019174    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:25:33.023619 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:33.023580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" event={"ID":"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a","Type":"ContainerStarted","Data":"0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc"}
Jul 17 19:25:33.425822 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:33.425777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:33.426073 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:33.426050    2563 patch_prober.go:28] interesting pod/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 17 19:25:33.426171 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:33.426148    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 17 19:25:34.425523 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:34.425483    2563 patch_prober.go:28] interesting pod/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 17 19:25:34.425924 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:34.425565    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 17 19:25:35.425760 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:35.425722    2563 patch_prober.go:28] interesting pod/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 17 19:25:35.426257 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:35.425793    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 17 19:25:36.426274 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:36.426235    2563 patch_prober.go:28] interesting pod/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 17 19:25:36.426678 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:36.426291    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 17 19:25:37.428884 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:37.428854    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:37.429484 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:37.429209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:37.429826 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:37.429806    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:25:37.502807 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:37.502775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:37.601404 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:37.601369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:38.420083 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.420052    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"]
Jul 17 19:25:38.420339 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.420313    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq" podUID="f6f081c1-72bf-4c60-a3f3-a6b0ee280563" containerName="maas-api" containerID="cri-o://83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10" gracePeriod=30
Jul 17 19:25:38.459797 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.459759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:38.674279 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.674217    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:38.750399 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.750368    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-maas-api-tls\") pod \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") "
Jul 17 19:25:38.750579 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.750408    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-openshift-service-ca\") pod \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") "
Jul 17 19:25:38.750579 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.750446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2zrz\" (UniqueName: \"kubernetes.io/projected/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-kube-api-access-z2zrz\") pod \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\" (UID: \"f6f081c1-72bf-4c60-a3f3-a6b0ee280563\") "
Jul 17 19:25:38.750862 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.750834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6f081c1-72bf-4c60-a3f3-a6b0ee280563" (UID: "f6f081c1-72bf-4c60-a3f3-a6b0ee280563"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:38.752710 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.752689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6f081c1-72bf-4c60-a3f3-a6b0ee280563" (UID: "f6f081c1-72bf-4c60-a3f3-a6b0ee280563"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:25:38.752823 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.752806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-kube-api-access-z2zrz" (OuterVolumeSpecName: "kube-api-access-z2zrz") pod "f6f081c1-72bf-4c60-a3f3-a6b0ee280563" (UID: "f6f081c1-72bf-4c60-a3f3-a6b0ee280563"). InnerVolumeSpecName "kube-api-access-z2zrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:25:38.852259 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.852222    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:38.852259 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.852252    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:38.852259 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:38.852262    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zrz\" (UniqueName: \"kubernetes.io/projected/f6f081c1-72bf-4c60-a3f3-a6b0ee280563-kube-api-access-z2zrz\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:39.050953 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.050850    2563 generic.go:356] "Generic (PLEG): container finished" podID="f6f081c1-72bf-4c60-a3f3-a6b0ee280563" containerID="83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10" exitCode=0
Jul 17 19:25:39.050953 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.050906    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"
Jul 17 19:25:39.050953 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.050942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq" event={"ID":"f6f081c1-72bf-4c60-a3f3-a6b0ee280563","Type":"ContainerDied","Data":"83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10"}
Jul 17 19:25:39.051254 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.050994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq" event={"ID":"f6f081c1-72bf-4c60-a3f3-a6b0ee280563","Type":"ContainerDied","Data":"06c56df47bb6ecf7ba5f0e683db9f0ee366c7b71cf128fe3bb9dd3628042a435"}
Jul 17 19:25:39.051254 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.051015    2563 scope.go:117] "RemoveContainer" containerID="83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10"
Jul 17 19:25:39.059909 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.059891    2563 scope.go:117] "RemoveContainer" containerID="83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10"
Jul 17 19:25:39.060203 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:25:39.060170    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10\": container with ID starting with 83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10 not found: ID does not exist" containerID="83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10"
Jul 17 19:25:39.060257 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.060211    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10"} err="failed to get container status \"83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10\": rpc error: code = NotFound desc = could not find container \"83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10\": container with ID starting with 83a524c4693812eb83182c2beb9d59246ba53b2f6ed9328bfefa0bf140204c10 not found: ID does not exist"
Jul 17 19:25:39.074594 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.074562    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"]
Jul 17 19:25:39.076631 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.076610    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40565f-98696c689-kktgq"]
Jul 17 19:25:39.699659 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:39.699621    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6f081c1-72bf-4c60-a3f3-a6b0ee280563" path="/var/lib/kubelet/pods/f6f081c1-72bf-4c60-a3f3-a6b0ee280563/volumes"
Jul 17 19:25:41.606428 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:41.606392    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:25:41.638175 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:41.638141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:41.738328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:41.738295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:43.443805 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:43.443768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:50.345624 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:50.345586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:50.483550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:50.483517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:51.106431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:51.106392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:54.123081 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:54.123048    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"]
Jul 17 19:25:54.123556 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:54.123346    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy" containerID="cri-o://4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c" gracePeriod=30
Jul 17 19:25:54.142457 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:54.142426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:54.343576 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:54.343543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:55.017648 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:55.017616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:55.120072 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:55.120039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:25:56.537374 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:56.537329    2563 patch_prober.go:28] interesting pod/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 19:25:56.537815 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:56.537427    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 19:25:59.374686 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.374663    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:25:59.429782 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csp8z\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-kube-api-access-csp8z\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.429782 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429740    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-token\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.429782 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429769    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099b2c46-c022-44a7-a208-c85b4f7f639a-istiod-ca-cert\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.430017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429794    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-data\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.430017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-socket\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.430017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-credential-socket\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.430017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-certs\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.430017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-envoy\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.430017 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.429938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-podinfo\") pod \"099b2c46-c022-44a7-a208-c85b4f7f639a\" (UID: \"099b2c46-c022-44a7-a208-c85b4f7f639a\") "
Jul 17 19:25:59.430350 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.430215    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099b2c46-c022-44a7-a208-c85b4f7f639a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:59.430350 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.430296    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:25:59.430350 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.430311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:25:59.430627 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.430588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:25:59.431013 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.430993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-data" (OuterVolumeSpecName: "istio-data") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:25:59.432230 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.432213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-token" (OuterVolumeSpecName: "istio-token") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:25:59.432389 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.432366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:25:59.432704 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.432678    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-kube-api-access-csp8z" (OuterVolumeSpecName: "kube-api-access-csp8z") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "kube-api-access-csp8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:25:59.432792 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.432719    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "099b2c46-c022-44a7-a208-c85b4f7f639a" (UID: "099b2c46-c022-44a7-a208-c85b4f7f639a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:25:59.531475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531434    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csp8z\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-kube-api-access-csp8z\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:59.531475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531470    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:59.531475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531479    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099b2c46-c022-44a7-a208-c85b4f7f639a-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:59.531475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531487    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:59.531735 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531496    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:59.531735 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531503    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:59.531735 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531512    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:59.531735 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531519    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:25:59.531735 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:25:59.531526    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099b2c46-c022-44a7-a208-c85b4f7f639a-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:00.142405 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.142368    2563 generic.go:356] "Generic (PLEG): container finished" podID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerID="4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c" exitCode=0
Jul 17 19:26:00.142582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.142442    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"
Jul 17 19:26:00.142582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.142457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" event={"ID":"099b2c46-c022-44a7-a208-c85b4f7f639a","Type":"ContainerDied","Data":"4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c"}
Jul 17 19:26:00.142582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.142516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj" event={"ID":"099b2c46-c022-44a7-a208-c85b4f7f639a","Type":"ContainerDied","Data":"1f7c3097b74188ee2ea465674ef93ea84778ff4414a31bb7c8b8c71c4f42fe89"}
Jul 17 19:26:00.142582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.142539    2563 scope.go:117] "RemoveContainer" containerID="4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c"
Jul 17 19:26:00.150716 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.150697    2563 scope.go:117] "RemoveContainer" containerID="4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7"
Jul 17 19:26:00.158334 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.158315    2563 scope.go:117] "RemoveContainer" containerID="4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c"
Jul 17 19:26:00.158595 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:00.158575    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c\": container with ID starting with 4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c not found: ID does not exist" containerID="4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c"
Jul 17 19:26:00.158642 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.158605    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c"} err="failed to get container status \"4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c\": rpc error: code = NotFound desc = could not find container \"4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c\": container with ID starting with 4b31da4f105c9956ddad9c1a11f7f5d749625516adc4ae632a3b3a2c435c811c not found: ID does not exist"
Jul 17 19:26:00.158642 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.158625    2563 scope.go:117] "RemoveContainer" containerID="4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7"
Jul 17 19:26:00.158872 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:00.158857    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7\": container with ID starting with 4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7 not found: ID does not exist" containerID="4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7"
Jul 17 19:26:00.158927 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.158875    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7"} err="failed to get container status \"4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7\": rpc error: code = NotFound desc = could not find container \"4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7\": container with ID starting with 4a8448a1f6931f7e42ab971ce4a0260402a8485c36100f5b104758b0b48e9da7 not found: ID does not exist"
Jul 17 19:26:00.162091 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.162068    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"]
Jul 17 19:26:00.165694 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:00.165675    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-40565f-openshift-default-78d9db77dc-6kpjj"]
Jul 17 19:26:01.693745 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:01.693711    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" path="/var/lib/kubelet/pods/099b2c46-c022-44a7-a208-c85b4f7f639a/volumes"
Jul 17 19:26:04.884206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:04.884173    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"]
Jul 17 19:26:04.884653 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:04.884382    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z" podUID="211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" containerName="maas-api" containerID="cri-o://5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26" gracePeriod=30
Jul 17 19:26:04.906801 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:04.906773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:05.127874 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.127849    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:26:05.167028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.166941    2563 generic.go:356] "Generic (PLEG): container finished" podID="211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" containerID="5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26" exitCode=0
Jul 17 19:26:05.167028 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.166997    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"
Jul 17 19:26:05.167250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.167020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z" event={"ID":"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1","Type":"ContainerDied","Data":"5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26"}
Jul 17 19:26:05.167250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.167077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z" event={"ID":"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1","Type":"ContainerDied","Data":"63a52014553fcc17d2e66349a8d5f7e64bc0e4b570ebec1a84ded90422858f02"}
Jul 17 19:26:05.167250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.167102    2563 scope.go:117] "RemoveContainer" containerID="5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26"
Jul 17 19:26:05.175203 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.175185    2563 scope.go:117] "RemoveContainer" containerID="5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26"
Jul 17 19:26:05.175497 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:05.175479    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26\": container with ID starting with 5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26 not found: ID does not exist" containerID="5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26"
Jul 17 19:26:05.175559 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.175506    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26"} err="failed to get container status \"5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26\": rpc error: code = NotFound desc = could not find container \"5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26\": container with ID starting with 5363a457eda4e0fc2b2196b8d0b4a28f1e53433d5ccadb41f67a8843e0a4cb26 not found: ID does not exist"
Jul 17 19:26:05.182801 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.182774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-openshift-service-ca\") pod \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") "
Jul 17 19:26:05.182898 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.182850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-maas-api-tls\") pod \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") "
Jul 17 19:26:05.182898 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.182892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fngv\" (UniqueName: \"kubernetes.io/projected/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-kube-api-access-5fngv\") pod \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\" (UID: \"211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1\") "
Jul 17 19:26:05.183174 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.183152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" (UID: "211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:26:05.184946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.184927    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" (UID: "211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:26:05.185019 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.184979    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-kube-api-access-5fngv" (OuterVolumeSpecName: "kube-api-access-5fngv") pod "211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" (UID: "211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1"). InnerVolumeSpecName "kube-api-access-5fngv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:26:05.283727 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.283692    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:05.283727 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.283723    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:05.283727 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.283733    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fngv\" (UniqueName: \"kubernetes.io/projected/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1-kube-api-access-5fngv\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:05.487646 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.487619    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"]
Jul 17 19:26:05.491326 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.491301    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-460276-c66895bf5-88h2z"]
Jul 17 19:26:05.698142 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:05.698071    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" path="/var/lib/kubelet/pods/211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1/volumes"
Jul 17 19:26:08.002057 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:08.002020    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:26:09.554324 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:09.554282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:16.702966 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:16.702929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:16.803814 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:16.803779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:20.225529 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:20.225490    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"]
Jul 17 19:26:20.225963 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:20.225707    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy" containerID="cri-o://0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc" gracePeriod=30
Jul 17 19:26:20.253338 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:20.253276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:20.496780 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:20.496648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:20.782308 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:20.782184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:21.094361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:21.094330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:21.623575 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:21.623543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:22.431748 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:22.431715    2563 patch_prober.go:28] interesting pod/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 19:26:22.431927 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:22.431771    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 19:26:24.381714 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.381643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:24.480397 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.480368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:24.483871 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.483843    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"]
Jul 17 19:26:24.484159 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484145    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" containerName="maas-api"
Jul 17 19:26:24.484206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484160    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" containerName="maas-api"
Jul 17 19:26:24.484206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484175    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy"
Jul 17 19:26:24.484206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484180    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy"
Jul 17 19:26:24.484206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484186    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f081c1-72bf-4c60-a3f3-a6b0ee280563" containerName="maas-api"
Jul 17 19:26:24.484206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484193    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f081c1-72bf-4c60-a3f3-a6b0ee280563" containerName="maas-api"
Jul 17 19:26:24.484206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484197    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy"
Jul 17 19:26:24.484206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484202    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy"
Jul 17 19:26:24.484206 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484208    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484213    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484224    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484229    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484273    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484280    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="211fc4f7-fe0c-4fc0-97c0-3b15bf67bcb1" containerName="maas-api"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484287    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f081c1-72bf-4c60-a3f3-a6b0ee280563" containerName="maas-api"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484293    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484299    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="099b2c46-c022-44a7-a208-c85b4f7f639a" containerName="istio-proxy"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484304    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:24.484466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.484309    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:24.487057 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.487042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:24.489750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.489729    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:26:24.489866 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.489855    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:26:24.490525 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.490510    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-98e305-serving-cert\""
Jul 17 19:26:24.490633 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.490605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:26:24.500952 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.500922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"]
Jul 17 19:26:24.548782 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.548742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrz72\" (UniqueName: \"kubernetes.io/projected/773daa09-4b25-466b-b70e-9b5d0d6e28e5-kube-api-access-vrz72\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:24.548981 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.548853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:24.548981 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.548885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/773daa09-4b25-466b-b70e-9b5d0d6e28e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:24.581486 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.581444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:24.649409 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.649326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrz72\" (UniqueName: \"kubernetes.io/projected/773daa09-4b25-466b-b70e-9b5d0d6e28e5-kube-api-access-vrz72\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:24.649581 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.649413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:24.649581 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.649437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/773daa09-4b25-466b-b70e-9b5d0d6e28e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:24.649700 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:24.649576    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-serving-cert: secret "maas-api-e2e-shared-a-98e305-serving-cert" not found
Jul 17 19:26:24.649700 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:24.649678    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls podName:773daa09-4b25-466b-b70e-9b5d0d6e28e5 nodeName:}" failed. No retries permitted until 2026-07-17 19:26:25.149655561 +0000 UTC m=+3194.060482319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls") pod "maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj" (UID: "773daa09-4b25-466b-b70e-9b5d0d6e28e5") : secret "maas-api-e2e-shared-a-98e305-serving-cert" not found
Jul 17 19:26:24.650065 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.650043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/773daa09-4b25-466b-b70e-9b5d0d6e28e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:24.658492 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:24.658473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrz72\" (UniqueName: \"kubernetes.io/projected/773daa09-4b25-466b-b70e-9b5d0d6e28e5-kube-api-access-vrz72\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:25.108337 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.108303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:25.153582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.153549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:25.156211 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.156175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:25.208595 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.208540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:25.397446 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.397418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:25.476973 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.476947    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:26:25.546225 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.546192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"]
Jul 17 19:26:25.549340 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:26:25.549313    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773daa09_4b25_466b_b70e_9b5d0d6e28e5.slice/crio-f551c234e10d3c7d532a1754a43769556d6008bc020e30ce11ba3da4dbc1fada WatchSource:0}: Error finding container f551c234e10d3c7d532a1754a43769556d6008bc020e30ce11ba3da4dbc1fada: Status 404 returned error can't find the container with id f551c234e10d3c7d532a1754a43769556d6008bc020e30ce11ba3da4dbc1fada
Jul 17 19:26:25.556042 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr7f8\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-kube-api-access-nr7f8\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556187 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istiod-ca-cert\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556187 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-data\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556187 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-socket\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-certs\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-credential-socket\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556343    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-podinfo\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:26:25.556413 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-token\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556684 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556463    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-envoy\") pod \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\" (UID: \"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a\") "
Jul 17 19:26:25.556684 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556532    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:26:25.556684 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:26:25.556839 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556731    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:25.556839 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556745    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:25.556839 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556755    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:25.556994 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.556892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:26:25.557106 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.557021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:26:25.558934 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.558908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:26:25.558934 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.558909    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-kube-api-access-nr7f8" (OuterVolumeSpecName: "kube-api-access-nr7f8") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "kube-api-access-nr7f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:26:25.559047 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.558911    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:26:25.559047 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.558992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" (UID: "9b27a5a2-a997-422d-8e9a-6fc05c71bb6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:26:25.658250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.658163    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr7f8\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-kube-api-access-nr7f8\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:25.658250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.658194    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:25.658250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.658208    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:25.658250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.658223    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:25.658250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.658234    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:25.658250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:25.658245    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:26.086676 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.086643    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"]
Jul 17 19:26:26.086926 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.086914    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:26.086969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.086927    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f38c99-9852-4bee-a01f-16cd289868dd" containerName="revoke-keys"
Jul 17 19:26:26.086969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.086950    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy"
Jul 17 19:26:26.086969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.086955    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy"
Jul 17 19:26:26.087062 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.087001    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy"
Jul 17 19:26:26.089844 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.089821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"
Jul 17 19:26:26.094952 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.094931    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:26:26.112804 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.112779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"]
Jul 17 19:26:26.163055 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.163014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8\" (UID: \"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"
Jul 17 19:26:26.258452 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.258364    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerID="0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc" exitCode=0
Jul 17 19:26:26.258452 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.258414    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"
Jul 17 19:26:26.258452 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.258444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" event={"ID":"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a","Type":"ContainerDied","Data":"0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc"}
Jul 17 19:26:26.258722 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.258493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx" event={"ID":"9b27a5a2-a997-422d-8e9a-6fc05c71bb6a","Type":"ContainerDied","Data":"ad57eb896789ab64b41c185f5df8550015d0942a489e7992db75c1926c604309"}
Jul 17 19:26:26.258722 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.258517    2563 scope.go:117] "RemoveContainer" containerID="0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc"
Jul 17 19:26:26.260003 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.259983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj" event={"ID":"773daa09-4b25-466b-b70e-9b5d0d6e28e5","Type":"ContainerStarted","Data":"20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84"}
Jul 17 19:26:26.260146 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.260018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj" event={"ID":"773daa09-4b25-466b-b70e-9b5d0d6e28e5","Type":"ContainerStarted","Data":"f551c234e10d3c7d532a1754a43769556d6008bc020e30ce11ba3da4dbc1fada"}
Jul 17 19:26:26.260228 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.260141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:26.263572 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.263551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8\" (UID: \"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"
Jul 17 19:26:26.264135 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.264095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8\" (UID: \"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"
Jul 17 19:26:26.266865 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.266847    2563 scope.go:117] "RemoveContainer" containerID="49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561"
Jul 17 19:26:26.274418 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.274399    2563 scope.go:117] "RemoveContainer" containerID="0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc"
Jul 17 19:26:26.274706 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:26.274687    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc\": container with ID starting with 0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc not found: ID does not exist" containerID="0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc"
Jul 17 19:26:26.274784 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.274719    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc"} err="failed to get container status \"0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc\": rpc error: code = NotFound desc = could not find container \"0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc\": container with ID starting with 0287681b0a16588ccdd897f5c1889053bc517e10b3bf4001de2c2ff040ac42cc not found: ID does not exist"
Jul 17 19:26:26.274784 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.274744    2563 scope.go:117] "RemoveContainer" containerID="49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561"
Jul 17 19:26:26.275007 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:26.274990    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561\": container with ID starting with 49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561 not found: ID does not exist" containerID="49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561"
Jul 17 19:26:26.275047 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.275014    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561"} err="failed to get container status \"49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561\": rpc error: code = NotFound desc = could not find container \"49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561\": container with ID starting with 49f8bd0c56ab27c89529ac6058a19c6d1602a290ae2f2cdf626e46012d8b5561 not found: ID does not exist"
Jul 17 19:26:26.284563 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.284519    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj" podStartSLOduration=1.833009706 podStartE2EDuration="2.284505405s" totalImagesPullingTime="451.495699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:26:26.283508365 +0000 UTC m=+3195.194335145" watchObservedRunningTime="2026-07-17 19:26:26.284505405 +0000 UTC m=+3195.195332165"
Jul 17 19:26:26.298172 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.298144    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"]
Jul 17 19:26:26.300823 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.300803    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-460276-openshift-default-9d4ff4587-w5lcx"]
Jul 17 19:26:26.399724 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.399688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"
Jul 17 19:26:26.554087 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:26.554013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"]
Jul 17 19:26:26.557277 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:26:26.557249    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab06e180_4f1b_4d3c_a0e0_a84a7965c7e7.slice/crio-b917dbdcfd20f0514afcb835d7ac09a366f520390da807d9cece3badd655664b WatchSource:0}: Error finding container b917dbdcfd20f0514afcb835d7ac09a366f520390da807d9cece3badd655664b: Status 404 returned error can't find the container with id b917dbdcfd20f0514afcb835d7ac09a366f520390da807d9cece3badd655664b
Jul 17 19:26:27.265599 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:27.265560    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerID="585fd3f732c54dc9a2c393a61c2c8baf6d5bf3115808ef318d38272154fb6048" exitCode=6
Jul 17 19:26:27.265789 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:27.265648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" event={"ID":"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7","Type":"ContainerDied","Data":"585fd3f732c54dc9a2c393a61c2c8baf6d5bf3115808ef318d38272154fb6048"}
Jul 17 19:26:27.265789 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:27.265698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" event={"ID":"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7","Type":"ContainerStarted","Data":"b917dbdcfd20f0514afcb835d7ac09a366f520390da807d9cece3badd655664b"}
Jul 17 19:26:27.265920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:27.265800    2563 scope.go:117] "RemoveContainer" containerID="585fd3f732c54dc9a2c393a61c2c8baf6d5bf3115808ef318d38272154fb6048"
Jul 17 19:26:27.693756 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:27.693722    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" path="/var/lib/kubelet/pods/9b27a5a2-a997-422d-8e9a-6fc05c71bb6a/volumes"
Jul 17 19:26:28.273421 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:28.273322    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerID="881e00fea2e39467621697aaaf129cc29cedd65d196ede37a957b54d98765d20" exitCode=6
Jul 17 19:26:28.273421 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:28.273403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" event={"ID":"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7","Type":"ContainerDied","Data":"881e00fea2e39467621697aaaf129cc29cedd65d196ede37a957b54d98765d20"}
Jul 17 19:26:28.273663 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:28.273457    2563 scope.go:117] "RemoveContainer" containerID="585fd3f732c54dc9a2c393a61c2c8baf6d5bf3115808ef318d38272154fb6048"
Jul 17 19:26:28.273663 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:28.273627    2563 scope.go:117] "RemoveContainer" containerID="881e00fea2e39467621697aaaf129cc29cedd65d196ede37a957b54d98765d20"
Jul 17 19:26:28.273897 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:28.273874    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8_odh-ai-gateway-infra(ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7"
Jul 17 19:26:28.886342 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:28.886311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:28.900362 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:28.900320    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:26:29.210289 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.210202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:29.279247 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.279218    2563 scope.go:117] "RemoveContainer" containerID="881e00fea2e39467621697aaaf129cc29cedd65d196ede37a957b54d98765d20"
Jul 17 19:26:29.279413 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:29.279393    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8_odh-ai-gateway-infra(ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7"
Jul 17 19:26:29.502722 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.502638    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"]
Jul 17 19:26:29.502971 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.502958    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy"
Jul 17 19:26:29.503009 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.502972    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy"
Jul 17 19:26:29.503041 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.503032    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b27a5a2-a997-422d-8e9a-6fc05c71bb6a" containerName="istio-proxy"
Jul 17 19:26:29.506931 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.506914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.509479 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.509455    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-2109fd-openshift-default-dockercfg-dfgvl\""
Jul 17 19:26:29.516860 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.516830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"]
Jul 17 19:26:29.525923 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.525873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:29.593341 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w24t\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-kube-api-access-7w24t\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.593549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-envoy\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.593549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-socket\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.593549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-token\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.593549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-podinfo\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.593549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-data\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.593549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-credential-socket\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.593549 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-certs\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.593928 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.593557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df955cae-ce35-484c-9af0-a81c1e13fc5d-istiod-ca-cert\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.694656 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-envoy\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.694855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-socket\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.694855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-token\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.694855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-podinfo\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.694855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-data\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.694855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-credential-socket\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.694855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-certs\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.694855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df955cae-ce35-484c-9af0-a81c1e13fc5d-istiod-ca-cert\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.695242 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.694904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w24t\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-kube-api-access-7w24t\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.695242 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.695158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-socket\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.695242 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.695215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-data\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.695367 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.695261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-certs\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.695581 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.695563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df955cae-ce35-484c-9af0-a81c1e13fc5d-istiod-ca-cert\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.695622 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.695522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-credential-socket\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.697177 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.697157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-envoy\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.697407 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.697389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-podinfo\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.704040 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.704005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w24t\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-kube-api-access-7w24t\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.704270 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.704243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-token\") pod \"e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") " pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.817728 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.817640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:29.960229 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.960203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"]
Jul 17 19:26:29.962936 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:26:29.962887    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf955cae_ce35_484c_9af0_a81c1e13fc5d.slice/crio-fc00c33c42dd9cc762494f40fc61ef9eef3c48755d1d86d2068b54f0faa522bc WatchSource:0}: Error finding container fc00c33c42dd9cc762494f40fc61ef9eef3c48755d1d86d2068b54f0faa522bc: Status 404 returned error can't find the container with id fc00c33c42dd9cc762494f40fc61ef9eef3c48755d1d86d2068b54f0faa522bc
Jul 17 19:26:29.965069 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.965038    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:26:29.965157 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.965109    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:26:29.965194 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:29.965164    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:26:30.007992 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:30.007967    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:30.286057 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:30.286011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" event={"ID":"df955cae-ce35-484c-9af0-a81c1e13fc5d","Type":"ContainerStarted","Data":"f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92"}
Jul 17 19:26:30.286256 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:30.286077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" event={"ID":"df955cae-ce35-484c-9af0-a81c1e13fc5d","Type":"ContainerStarted","Data":"fc00c33c42dd9cc762494f40fc61ef9eef3c48755d1d86d2068b54f0faa522bc"}
Jul 17 19:26:30.308419 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:30.308373    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" podStartSLOduration=1.308355428 podStartE2EDuration="1.308355428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:26:30.307010344 +0000 UTC m=+3199.217837116" watchObservedRunningTime="2026-07-17 19:26:30.308355428 +0000 UTC m=+3199.219182189"
Jul 17 19:26:30.818284 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:30.818246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:31.823469 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:31.823436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:32.221864 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.221832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:32.271858 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.271825    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:26:32.294667 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.294635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:32.295685 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.295659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:32.295954 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.295936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:32.406016 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.405985    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"]
Jul 17 19:26:32.409421 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.409394    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.411836 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.411815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-2109fd-serving-cert\""
Jul 17 19:26:32.418759 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.418733    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"]
Jul 17 19:26:32.426284 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.426255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:32.521621 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.521510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9292d38-97ce-4073-9841-00123362c521-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.521779 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.521670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.521779 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.521733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hpj\" (UniqueName: \"kubernetes.io/projected/f9292d38-97ce-4073-9841-00123362c521-kube-api-access-69hpj\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.623219 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.623180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.623391 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.623228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hpj\" (UniqueName: \"kubernetes.io/projected/f9292d38-97ce-4073-9841-00123362c521-kube-api-access-69hpj\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.623391 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.623308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9292d38-97ce-4073-9841-00123362c521-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.623391 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:32.623326    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-serving-cert: secret "maas-api-e2e-shared-b-2109fd-serving-cert" not found
Jul 17 19:26:32.623502 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:32.623420    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls podName:f9292d38-97ce-4073-9841-00123362c521 nodeName:}" failed. No retries permitted until 2026-07-17 19:26:33.123403119 +0000 UTC m=+3202.034229858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls") pod "maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg" (UID: "f9292d38-97ce-4073-9841-00123362c521") : secret "maas-api-e2e-shared-b-2109fd-serving-cert" not found
Jul 17 19:26:32.623885 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.623868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9292d38-97ce-4073-9841-00123362c521-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.635798 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.635772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hpj\" (UniqueName: \"kubernetes.io/projected/f9292d38-97ce-4073-9841-00123362c521-kube-api-access-69hpj\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:32.912447 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:32.912415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:33.009716 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:33.009668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:33.128192 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:33.128150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:33.130720 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:33.130694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls\") pod \"maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:33.321029 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:33.320934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:33.455988 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:33.455872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"]
Jul 17 19:26:33.463203 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:26:33.463168    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9292d38_97ce_4073_9841_00123362c521.slice/crio-dc80df21149f36cbebbc4eb4686d624081dd9f892a65d262dcfd983426025120 WatchSource:0}: Error finding container dc80df21149f36cbebbc4eb4686d624081dd9f892a65d262dcfd983426025120: Status 404 returned error can't find the container with id dc80df21149f36cbebbc4eb4686d624081dd9f892a65d262dcfd983426025120
Jul 17 19:26:34.305625 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:34.305571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg" event={"ID":"f9292d38-97ce-4073-9841-00123362c521","Type":"ContainerStarted","Data":"5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d"}
Jul 17 19:26:34.306065 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:34.305639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg" event={"ID":"f9292d38-97ce-4073-9841-00123362c521","Type":"ContainerStarted","Data":"dc80df21149f36cbebbc4eb4686d624081dd9f892a65d262dcfd983426025120"}
Jul 17 19:26:34.323464 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:34.323411    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg" podStartSLOduration=2.028465127 podStartE2EDuration="2.323395575s" totalImagesPullingTime="294.930448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:26:34.322454013 +0000 UTC m=+3203.233280789" watchObservedRunningTime="2026-07-17 19:26:34.323395575 +0000 UTC m=+3203.234222335"
Jul 17 19:26:35.310807 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:35.310777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:36.720499 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:36.720449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:38.205810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:38.205771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:26:39.695005 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:39.689601    2563 scope.go:117] "RemoveContainer" containerID="881e00fea2e39467621697aaaf129cc29cedd65d196ede37a957b54d98765d20"
Jul 17 19:26:40.334224 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:40.334108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" event={"ID":"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7","Type":"ContainerStarted","Data":"3e1f5673b9a32ecb1869bf98a068fff954a0f7c3a55b89b80a94e84bf0f4dd06"}
Jul 17 19:26:40.351258 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:40.351202    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" podStartSLOduration=14.351184398000001 podStartE2EDuration="14.351184398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:26:40.350236729 +0000 UTC m=+3209.261063509" watchObservedRunningTime="2026-07-17 19:26:40.351184398 +0000 UTC m=+3209.262011158"
Jul 17 19:26:41.321235 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:41.321205    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:26:41.341250 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:41.341219    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerID="3e1f5673b9a32ecb1869bf98a068fff954a0f7c3a55b89b80a94e84bf0f4dd06" exitCode=6
Jul 17 19:26:41.341414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:41.341282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" event={"ID":"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7","Type":"ContainerDied","Data":"3e1f5673b9a32ecb1869bf98a068fff954a0f7c3a55b89b80a94e84bf0f4dd06"}
Jul 17 19:26:41.341414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:41.341329    2563 scope.go:117] "RemoveContainer" containerID="881e00fea2e39467621697aaaf129cc29cedd65d196ede37a957b54d98765d20"
Jul 17 19:26:41.341623 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:41.341608    2563 scope.go:117] "RemoveContainer" containerID="3e1f5673b9a32ecb1869bf98a068fff954a0f7c3a55b89b80a94e84bf0f4dd06"
Jul 17 19:26:41.341822 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:26:41.341803    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8_odh-ai-gateway-infra(ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7"
Jul 17 19:26:41.359085 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:41.359055    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"]
Jul 17 19:26:42.523336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:42.523307    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"
Jul 17 19:26:42.617606 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:42.617519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7-maas-api-service-ca\") pod \"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7\" (UID: \"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7\") "
Jul 17 19:26:42.617858 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:42.617834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" (UID: "ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:26:42.718339 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:42.718287    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:26:43.353060 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:43.353033    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"
Jul 17 19:26:43.353060 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:43.353048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8" event={"ID":"ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7","Type":"ContainerDied","Data":"b917dbdcfd20f0514afcb835d7ac09a366f520390da807d9cece3badd655664b"}
Jul 17 19:26:43.353331 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:43.353105    2563 scope.go:117] "RemoveContainer" containerID="3e1f5673b9a32ecb1869bf98a068fff954a0f7c3a55b89b80a94e84bf0f4dd06"
Jul 17 19:26:43.384393 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:43.384346    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"]
Jul 17 19:26:43.392820 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:43.392781    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jclg8"]
Jul 17 19:26:43.697703 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:43.697620    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" path="/var/lib/kubelet/pods/ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7/volumes"
Jul 17 19:26:46.973875 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:46.973835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:47.074277 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.074242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:47.296597 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.296505    2563 patch_prober.go:28] interesting pod/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 17 19:26:47.296597 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.296564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 17 19:26:47.372238 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.372205    2563 generic.go:356] "Generic (PLEG): container finished" podID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerID="f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92" exitCode=0
Jul 17 19:26:47.372437 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.372288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" event={"ID":"df955cae-ce35-484c-9af0-a81c1e13fc5d","Type":"ContainerDied","Data":"f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92"}
Jul 17 19:26:47.372745 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.372728    2563 scope.go:117] "RemoveContainer" containerID="f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92"
Jul 17 19:26:47.373766 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.373737    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:26:47.373855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.373805    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:26:47.373855 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:47.373835    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 19:26:48.381723 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:48.381678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" event={"ID":"df955cae-ce35-484c-9af0-a81c1e13fc5d","Type":"ContainerStarted","Data":"56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89"}
Jul 17 19:26:48.818382 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:48.818348    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:48.818696 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:48.818672    2563 patch_prober.go:28] interesting pod/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 17 19:26:48.818794 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:48.818726    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 17 19:26:49.819048 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:49.819014    2563 patch_prober.go:28] interesting pod/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 17 19:26:49.819452 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:49.819085    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 17 19:26:49.829868 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:49.829843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:50.818410 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:50.818371    2563 patch_prober.go:28] interesting pod/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 17 19:26:50.818582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:50.818427    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 17 19:26:51.819010 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:51.818971    2563 patch_prober.go:28] interesting pod/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Jul 17 19:26:51.819427 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:51.819030    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Jul 17 19:26:52.822516 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:52.822480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:52.822997 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:52.822746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:52.823479 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:52.823461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:26:53.202318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:53.202281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:26:53.740738 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:53.740705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:26:53.840732 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:26:53.840693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:27:13.270576 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:13.270534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:27:16.124101 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.124056    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"]
Jul 17 19:27:16.124601 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.124517    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:16.124601 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.124536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:16.124601 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.124571    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:16.124601 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.124579    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:16.124811 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.124651    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:16.124811 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.124666    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:16.124811 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.124676    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:16.129073 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.129052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"
Jul 17 19:27:16.131865 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.131844    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:27:16.134369 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.134346    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"]
Jul 17 19:27:16.213109 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.213069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18501c34-6065-4638-9604-b673a89be5d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w\" (UID: \"18501c34-6065-4638-9604-b673a89be5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"
Jul 17 19:27:16.314060 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.314021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18501c34-6065-4638-9604-b673a89be5d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w\" (UID: \"18501c34-6065-4638-9604-b673a89be5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"
Jul 17 19:27:16.314692 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.314671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18501c34-6065-4638-9604-b673a89be5d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w\" (UID: \"18501c34-6065-4638-9604-b673a89be5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"
Jul 17 19:27:16.411781 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.411692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:27:16.441605 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.441572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"
Jul 17 19:27:16.575893 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:16.575855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"]
Jul 17 19:27:16.579063 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:27:16.579023    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18501c34_6065_4638_9604_b673a89be5d3.slice/crio-a5ea2e5b0392bb4537f7086d7595f7cd73c40349857016a2ab029dc59f8eda40 WatchSource:0}: Error finding container a5ea2e5b0392bb4537f7086d7595f7cd73c40349857016a2ab029dc59f8eda40: Status 404 returned error can't find the container with id a5ea2e5b0392bb4537f7086d7595f7cd73c40349857016a2ab029dc59f8eda40
Jul 17 19:27:17.512544 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:17.512511    2563 generic.go:356] "Generic (PLEG): container finished" podID="18501c34-6065-4638-9604-b673a89be5d3" containerID="59876b38abbbbc3d9667a6551132ebc0718d8be54dedfbd469d799722fce46fb" exitCode=6
Jul 17 19:27:17.512961 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:17.512577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w" event={"ID":"18501c34-6065-4638-9604-b673a89be5d3","Type":"ContainerDied","Data":"59876b38abbbbc3d9667a6551132ebc0718d8be54dedfbd469d799722fce46fb"}
Jul 17 19:27:17.512961 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:17.512617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w" event={"ID":"18501c34-6065-4638-9604-b673a89be5d3","Type":"ContainerStarted","Data":"a5ea2e5b0392bb4537f7086d7595f7cd73c40349857016a2ab029dc59f8eda40"}
Jul 17 19:27:17.512961 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:17.512783    2563 scope.go:117] "RemoveContainer" containerID="59876b38abbbbc3d9667a6551132ebc0718d8be54dedfbd469d799722fce46fb"
Jul 17 19:27:18.519367 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:18.519273    2563 generic.go:356] "Generic (PLEG): container finished" podID="18501c34-6065-4638-9604-b673a89be5d3" containerID="3fc2820796bfc040a375d419c324898bce1685b912ce5d9b3bd236cac5a287cc" exitCode=6
Jul 17 19:27:18.519367 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:18.519331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w" event={"ID":"18501c34-6065-4638-9604-b673a89be5d3","Type":"ContainerDied","Data":"3fc2820796bfc040a375d419c324898bce1685b912ce5d9b3bd236cac5a287cc"}
Jul 17 19:27:18.519979 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:18.519378    2563 scope.go:117] "RemoveContainer" containerID="59876b38abbbbc3d9667a6551132ebc0718d8be54dedfbd469d799722fce46fb"
Jul 17 19:27:18.519979 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:18.519596    2563 scope.go:117] "RemoveContainer" containerID="3fc2820796bfc040a375d419c324898bce1685b912ce5d9b3bd236cac5a287cc"
Jul 17 19:27:18.519979 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:18.519852    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w_odh-ai-gateway-infra(18501c34-6065-4638-9604-b673a89be5d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w" podUID="18501c34-6065-4638-9604-b673a89be5d3"
Jul 17 19:27:19.526109 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:19.526081    2563 scope.go:117] "RemoveContainer" containerID="3fc2820796bfc040a375d419c324898bce1685b912ce5d9b3bd236cac5a287cc"
Jul 17 19:27:19.526517 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:19.526287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w_odh-ai-gateway-infra(18501c34-6065-4638-9604-b673a89be5d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w" podUID="18501c34-6065-4638-9604-b673a89be5d3"
Jul 17 19:27:22.584673 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:22.584640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:27:24.332207 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.332171    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"]
Jul 17 19:27:24.332702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.332639    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:24.332702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.332658    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab06e180-4f1b-4d3c-a0e0-a84a7965c7e7" containerName="revoke-keys"
Jul 17 19:27:24.335672 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.335651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"
Jul 17 19:27:24.342033 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.342006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"]
Jul 17 19:27:24.483330 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.483285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938cf889-47d2-4f9e-88c7-86f6d18a55a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr\" (UID: \"938cf889-47d2-4f9e-88c7-86f6d18a55a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"
Jul 17 19:27:24.584793 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.584709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938cf889-47d2-4f9e-88c7-86f6d18a55a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr\" (UID: \"938cf889-47d2-4f9e-88c7-86f6d18a55a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"
Jul 17 19:27:24.585366 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.585345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938cf889-47d2-4f9e-88c7-86f6d18a55a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr\" (UID: \"938cf889-47d2-4f9e-88c7-86f6d18a55a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"
Jul 17 19:27:24.647632 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.647600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"
Jul 17 19:27:24.776887 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:24.776799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"]
Jul 17 19:27:24.779651 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:27:24.779622    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938cf889_47d2_4f9e_88c7_86f6d18a55a1.slice/crio-6903cc8b4bff94fd5eb460faf15c703c29943a82eda1364b062f1ddccbfd4ee8 WatchSource:0}: Error finding container 6903cc8b4bff94fd5eb460faf15c703c29943a82eda1364b062f1ddccbfd4ee8: Status 404 returned error can't find the container with id 6903cc8b4bff94fd5eb460faf15c703c29943a82eda1364b062f1ddccbfd4ee8
Jul 17 19:27:25.552539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:25.552497    2563 generic.go:356] "Generic (PLEG): container finished" podID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerID="ba842b488a6782217908ac02c8fcce24fd80ddcdc76a5b93b77a48f7de45e5b9" exitCode=6
Jul 17 19:27:25.552946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:25.552589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr" event={"ID":"938cf889-47d2-4f9e-88c7-86f6d18a55a1","Type":"ContainerDied","Data":"ba842b488a6782217908ac02c8fcce24fd80ddcdc76a5b93b77a48f7de45e5b9"}
Jul 17 19:27:25.552946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:25.552687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr" event={"ID":"938cf889-47d2-4f9e-88c7-86f6d18a55a1","Type":"ContainerStarted","Data":"6903cc8b4bff94fd5eb460faf15c703c29943a82eda1364b062f1ddccbfd4ee8"}
Jul 17 19:27:25.552946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:25.552798    2563 scope.go:117] "RemoveContainer" containerID="ba842b488a6782217908ac02c8fcce24fd80ddcdc76a5b93b77a48f7de45e5b9"
Jul 17 19:27:26.559606 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:26.559522    2563 generic.go:356] "Generic (PLEG): container finished" podID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerID="1c07b80ab64256696cd8997bf9704949a33f529bad77903b6a3b679322809eed" exitCode=6
Jul 17 19:27:26.559606 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:26.559567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr" event={"ID":"938cf889-47d2-4f9e-88c7-86f6d18a55a1","Type":"ContainerDied","Data":"1c07b80ab64256696cd8997bf9704949a33f529bad77903b6a3b679322809eed"}
Jul 17 19:27:26.560092 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:26.559628    2563 scope.go:117] "RemoveContainer" containerID="ba842b488a6782217908ac02c8fcce24fd80ddcdc76a5b93b77a48f7de45e5b9"
Jul 17 19:27:26.560092 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:26.559874    2563 scope.go:117] "RemoveContainer" containerID="1c07b80ab64256696cd8997bf9704949a33f529bad77903b6a3b679322809eed"
Jul 17 19:27:26.560092 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:26.560075    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr_odh-ai-gateway-infra(938cf889-47d2-4f9e-88c7-86f6d18a55a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1"
Jul 17 19:27:27.566045 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:27.566013    2563 scope.go:117] "RemoveContainer" containerID="1c07b80ab64256696cd8997bf9704949a33f529bad77903b6a3b679322809eed"
Jul 17 19:27:27.566466 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:27.566238    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr_odh-ai-gateway-infra(938cf889-47d2-4f9e-88c7-86f6d18a55a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1"
Jul 17 19:27:32.689466 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:32.689430    2563 scope.go:117] "RemoveContainer" containerID="3fc2820796bfc040a375d419c324898bce1685b912ce5d9b3bd236cac5a287cc"
Jul 17 19:27:33.210946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.210913    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"]
Jul 17 19:27:33.221555 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.221250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"
Jul 17 19:27:33.221555 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.221311    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"]
Jul 17 19:27:33.260742 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.260661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0c665f-6bfb-457c-96d9-6420336fd7eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8\" (UID: \"da0c665f-6bfb-457c-96d9-6420336fd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"
Jul 17 19:27:33.361105 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.361063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0c665f-6bfb-457c-96d9-6420336fd7eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8\" (UID: \"da0c665f-6bfb-457c-96d9-6420336fd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"
Jul 17 19:27:33.361756 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.361736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0c665f-6bfb-457c-96d9-6420336fd7eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8\" (UID: \"da0c665f-6bfb-457c-96d9-6420336fd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"
Jul 17 19:27:33.533295 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.533199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"
Jul 17 19:27:33.594795 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.594761    2563 generic.go:356] "Generic (PLEG): container finished" podID="18501c34-6065-4638-9604-b673a89be5d3" containerID="646db70b8c5e5d9ffd7106bb728b608c16a58534fd0b8e5c51af8557fb217c64" exitCode=6
Jul 17 19:27:33.594917 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.594824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w" event={"ID":"18501c34-6065-4638-9604-b673a89be5d3","Type":"ContainerDied","Data":"646db70b8c5e5d9ffd7106bb728b608c16a58534fd0b8e5c51af8557fb217c64"}
Jul 17 19:27:33.594917 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.594870    2563 scope.go:117] "RemoveContainer" containerID="3fc2820796bfc040a375d419c324898bce1685b912ce5d9b3bd236cac5a287cc"
Jul 17 19:27:33.595330 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.595298    2563 scope.go:117] "RemoveContainer" containerID="646db70b8c5e5d9ffd7106bb728b608c16a58534fd0b8e5c51af8557fb217c64"
Jul 17 19:27:33.595585 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:33.595557    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w_odh-ai-gateway-infra(18501c34-6065-4638-9604-b673a89be5d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w" podUID="18501c34-6065-4638-9604-b673a89be5d3"
Jul 17 19:27:33.671954 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:33.671929    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"]
Jul 17 19:27:33.674329 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:27:33.674302    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0c665f_6bfb_457c_96d9_6420336fd7eb.slice/crio-4a8c486fe7a365e1db314a3ea414db3ae72b3e89cba9f107547756ad4e636589 WatchSource:0}: Error finding container 4a8c486fe7a365e1db314a3ea414db3ae72b3e89cba9f107547756ad4e636589: Status 404 returned error can't find the container with id 4a8c486fe7a365e1db314a3ea414db3ae72b3e89cba9f107547756ad4e636589
Jul 17 19:27:34.602037 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.601994    2563 generic.go:356] "Generic (PLEG): container finished" podID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerID="ec9dabeedb863c8809c6058fd85e1bafbaac89aa6b087d7ad23907c542efc189" exitCode=6
Jul 17 19:27:34.602449 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.602045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8" event={"ID":"da0c665f-6bfb-457c-96d9-6420336fd7eb","Type":"ContainerDied","Data":"ec9dabeedb863c8809c6058fd85e1bafbaac89aa6b087d7ad23907c542efc189"}
Jul 17 19:27:34.602449 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.602082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8" event={"ID":"da0c665f-6bfb-457c-96d9-6420336fd7eb","Type":"ContainerStarted","Data":"4a8c486fe7a365e1db314a3ea414db3ae72b3e89cba9f107547756ad4e636589"}
Jul 17 19:27:34.602449 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.602272    2563 scope.go:117] "RemoveContainer" containerID="ec9dabeedb863c8809c6058fd85e1bafbaac89aa6b087d7ad23907c542efc189"
Jul 17 19:27:34.622963 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.622933    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"]
Jul 17 19:27:34.756562 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.756537    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"
Jul 17 19:27:34.770418 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.770393    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18501c34-6065-4638-9604-b673a89be5d3-maas-api-service-ca\") pod \"18501c34-6065-4638-9604-b673a89be5d3\" (UID: \"18501c34-6065-4638-9604-b673a89be5d3\") "
Jul 17 19:27:34.770713 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.770692    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18501c34-6065-4638-9604-b673a89be5d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18501c34-6065-4638-9604-b673a89be5d3" (UID: "18501c34-6065-4638-9604-b673a89be5d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:27:34.871458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:34.871372    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18501c34-6065-4638-9604-b673a89be5d3-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:27:35.608261 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.608223    2563 generic.go:356] "Generic (PLEG): container finished" podID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerID="8de587a813f1c1e7e9e42655e82c0d77c69f2442de43802973bc087abc783cd5" exitCode=6
Jul 17 19:27:35.608750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.608312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8" event={"ID":"da0c665f-6bfb-457c-96d9-6420336fd7eb","Type":"ContainerDied","Data":"8de587a813f1c1e7e9e42655e82c0d77c69f2442de43802973bc087abc783cd5"}
Jul 17 19:27:35.608750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.608371    2563 scope.go:117] "RemoveContainer" containerID="ec9dabeedb863c8809c6058fd85e1bafbaac89aa6b087d7ad23907c542efc189"
Jul 17 19:27:35.608750 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.608547    2563 scope.go:117] "RemoveContainer" containerID="8de587a813f1c1e7e9e42655e82c0d77c69f2442de43802973bc087abc783cd5"
Jul 17 19:27:35.608750 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:35.608736    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8_odh-ai-gateway-infra(da0c665f-6bfb-457c-96d9-6420336fd7eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb"
Jul 17 19:27:35.609615 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.609598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"
Jul 17 19:27:35.609615 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.609611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w" event={"ID":"18501c34-6065-4638-9604-b673a89be5d3","Type":"ContainerDied","Data":"a5ea2e5b0392bb4537f7086d7595f7cd73c40349857016a2ab029dc59f8eda40"}
Jul 17 19:27:35.619284 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.619266    2563 scope.go:117] "RemoveContainer" containerID="646db70b8c5e5d9ffd7106bb728b608c16a58534fd0b8e5c51af8557fb217c64"
Jul 17 19:27:35.635754 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.635728    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"]
Jul 17 19:27:35.643274 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.643251    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-nqg2w"]
Jul 17 19:27:35.694269 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:35.694237    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18501c34-6065-4638-9604-b673a89be5d3" path="/var/lib/kubelet/pods/18501c34-6065-4638-9604-b673a89be5d3/volumes"
Jul 17 19:27:36.616316 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:36.616284    2563 scope.go:117] "RemoveContainer" containerID="8de587a813f1c1e7e9e42655e82c0d77c69f2442de43802973bc087abc783cd5"
Jul 17 19:27:36.616807 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:36.616530    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8_odh-ai-gateway-infra(da0c665f-6bfb-457c-96d9-6420336fd7eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb"
Jul 17 19:27:42.689003 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:42.688970    2563 scope.go:117] "RemoveContainer" containerID="1c07b80ab64256696cd8997bf9704949a33f529bad77903b6a3b679322809eed"
Jul 17 19:27:43.650072 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:43.650040    2563 generic.go:356] "Generic (PLEG): container finished" podID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerID="e5441593b8db4ea9d13e7cc410bd924feea7cbda6ad5744e22d9b25e7e1ccf1b" exitCode=6
Jul 17 19:27:43.650279 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:43.650109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr" event={"ID":"938cf889-47d2-4f9e-88c7-86f6d18a55a1","Type":"ContainerDied","Data":"e5441593b8db4ea9d13e7cc410bd924feea7cbda6ad5744e22d9b25e7e1ccf1b"}
Jul 17 19:27:43.650279 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:43.650193    2563 scope.go:117] "RemoveContainer" containerID="1c07b80ab64256696cd8997bf9704949a33f529bad77903b6a3b679322809eed"
Jul 17 19:27:43.650499 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:43.650482    2563 scope.go:117] "RemoveContainer" containerID="e5441593b8db4ea9d13e7cc410bd924feea7cbda6ad5744e22d9b25e7e1ccf1b"
Jul 17 19:27:43.650722 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:43.650700    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr_odh-ai-gateway-infra(938cf889-47d2-4f9e-88c7-86f6d18a55a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1"
Jul 17 19:27:44.676110 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:44.676065    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"]
Jul 17 19:27:44.809891 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:44.809864    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"
Jul 17 19:27:44.855629 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:44.855583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938cf889-47d2-4f9e-88c7-86f6d18a55a1-maas-api-service-ca\") pod \"938cf889-47d2-4f9e-88c7-86f6d18a55a1\" (UID: \"938cf889-47d2-4f9e-88c7-86f6d18a55a1\") "
Jul 17 19:27:44.855920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:44.855897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938cf889-47d2-4f9e-88c7-86f6d18a55a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "938cf889-47d2-4f9e-88c7-86f6d18a55a1" (UID: "938cf889-47d2-4f9e-88c7-86f6d18a55a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:27:44.956402 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:44.956307    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938cf889-47d2-4f9e-88c7-86f6d18a55a1-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:27:45.661553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:45.661523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr" event={"ID":"938cf889-47d2-4f9e-88c7-86f6d18a55a1","Type":"ContainerDied","Data":"6903cc8b4bff94fd5eb460faf15c703c29943a82eda1364b062f1ddccbfd4ee8"}
Jul 17 19:27:45.661553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:45.661539    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"
Jul 17 19:27:45.661800 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:45.661588    2563 scope.go:117] "RemoveContainer" containerID="e5441593b8db4ea9d13e7cc410bd924feea7cbda6ad5744e22d9b25e7e1ccf1b"
Jul 17 19:27:45.682235 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:45.682209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"]
Jul 17 19:27:45.688159 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:45.688137    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-xqdkr"]
Jul 17 19:27:45.693678 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:45.693655    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" path="/var/lib/kubelet/pods/938cf889-47d2-4f9e-88c7-86f6d18a55a1/volumes"
Jul 17 19:27:49.690055 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:49.690021    2563 scope.go:117] "RemoveContainer" containerID="8de587a813f1c1e7e9e42655e82c0d77c69f2442de43802973bc087abc783cd5"
Jul 17 19:27:50.685945 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:50.685860    2563 generic.go:356] "Generic (PLEG): container finished" podID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerID="9e2fef8441bcba7bbbc9cadf4ba676229b85c3e924aa711e7a07de68e31037e4" exitCode=6
Jul 17 19:27:50.685945 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:50.685909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8" event={"ID":"da0c665f-6bfb-457c-96d9-6420336fd7eb","Type":"ContainerDied","Data":"9e2fef8441bcba7bbbc9cadf4ba676229b85c3e924aa711e7a07de68e31037e4"}
Jul 17 19:27:50.686174 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:50.685963    2563 scope.go:117] "RemoveContainer" containerID="8de587a813f1c1e7e9e42655e82c0d77c69f2442de43802973bc087abc783cd5"
Jul 17 19:27:50.686277 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:50.686260    2563 scope.go:117] "RemoveContainer" containerID="9e2fef8441bcba7bbbc9cadf4ba676229b85c3e924aa711e7a07de68e31037e4"
Jul 17 19:27:50.686511 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:27:50.686490    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8_odh-ai-gateway-infra(da0c665f-6bfb-457c-96d9-6420336fd7eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb"
Jul 17 19:27:51.713022 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:51.712983    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"]
Jul 17 19:27:51.847222 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:51.847194    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"
Jul 17 19:27:51.917999 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:51.917964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0c665f-6bfb-457c-96d9-6420336fd7eb-maas-api-service-ca\") pod \"da0c665f-6bfb-457c-96d9-6420336fd7eb\" (UID: \"da0c665f-6bfb-457c-96d9-6420336fd7eb\") "
Jul 17 19:27:51.918385 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:51.918362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0c665f-6bfb-457c-96d9-6420336fd7eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da0c665f-6bfb-457c-96d9-6420336fd7eb" (UID: "da0c665f-6bfb-457c-96d9-6420336fd7eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:27:52.019437 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:52.019352    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0c665f-6bfb-457c-96d9-6420336fd7eb-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:27:52.699384 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:52.699352    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"
Jul 17 19:27:52.699384 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:52.699368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8" event={"ID":"da0c665f-6bfb-457c-96d9-6420336fd7eb","Type":"ContainerDied","Data":"4a8c486fe7a365e1db314a3ea414db3ae72b3e89cba9f107547756ad4e636589"}
Jul 17 19:27:52.699642 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:52.699422    2563 scope.go:117] "RemoveContainer" containerID="9e2fef8441bcba7bbbc9cadf4ba676229b85c3e924aa711e7a07de68e31037e4"
Jul 17 19:27:52.719768 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:52.719739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"]
Jul 17 19:27:52.725338 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:52.725314    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-tdgh8"]
Jul 17 19:27:53.695837 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:53.695801    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" path="/var/lib/kubelet/pods/da0c665f-6bfb-457c-96d9-6420336fd7eb/volumes"
Jul 17 19:27:53.867775 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:27:53.867738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:28:03.999714 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:03.999681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:28:11.755709 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:11.755679    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:28:11.763878 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:11.763848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:28:23.202062 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.202022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:28:23.254085 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254049    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"]
Jul 17 19:28:23.254417 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254403    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254420    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254434    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254440    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254447    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:23.254461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254453    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:23.254461 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254461    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254467    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254481    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254486    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254491    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254499    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254504    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254554    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254561    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254567    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254572    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254577    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254584    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="938cf889-47d2-4f9e-88c7-86f6d18a55a1" containerName="revoke-keys"
Jul 17 19:28:23.254645 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.254590    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:23.259294 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.258033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"
Jul 17 19:28:23.261429 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.261398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:28:23.265320 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.265291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"]
Jul 17 19:28:23.297811 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.297772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab789a6-3304-46c8-9d28-ce1b26371bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml\" (UID: \"9ab789a6-3304-46c8-9d28-ce1b26371bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"
Jul 17 19:28:23.398582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.398539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab789a6-3304-46c8-9d28-ce1b26371bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml\" (UID: \"9ab789a6-3304-46c8-9d28-ce1b26371bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"
Jul 17 19:28:23.399208 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.399187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab789a6-3304-46c8-9d28-ce1b26371bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml\" (UID: \"9ab789a6-3304-46c8-9d28-ce1b26371bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"
Jul 17 19:28:23.572020 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.571934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"
Jul 17 19:28:23.708734 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.708709    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"]
Jul 17 19:28:23.712264 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:28:23.712237    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab789a6_3304_46c8_9d28_ce1b26371bdf.slice/crio-c859c301c831b376f3ab049fe142c4fdacc8eb0baa61eb83aa02833ec3e8eb83 WatchSource:0}: Error finding container c859c301c831b376f3ab049fe142c4fdacc8eb0baa61eb83aa02833ec3e8eb83: Status 404 returned error can't find the container with id c859c301c831b376f3ab049fe142c4fdacc8eb0baa61eb83aa02833ec3e8eb83
Jul 17 19:28:23.834891 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:23.834803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml" event={"ID":"9ab789a6-3304-46c8-9d28-ce1b26371bdf","Type":"ContainerStarted","Data":"c859c301c831b376f3ab049fe142c4fdacc8eb0baa61eb83aa02833ec3e8eb83"}
Jul 17 19:28:24.378090 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.378052    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"]
Jul 17 19:28:24.378467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.378381    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:24.378467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.378392    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:24.378467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.378405    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:24.378467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.378410    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:24.378467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.378459    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="18501c34-6065-4638-9604-b673a89be5d3" containerName="revoke-keys"
Jul 17 19:28:24.378467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.378466    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da0c665f-6bfb-457c-96d9-6420336fd7eb" containerName="revoke-keys"
Jul 17 19:28:24.381247 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.381229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"
Jul 17 19:28:24.389957 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.389922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"]
Jul 17 19:28:24.407642 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.407615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c7b58e-7495-48f7-8716-999531e4e015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg\" (UID: \"85c7b58e-7495-48f7-8716-999531e4e015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"
Jul 17 19:28:24.508898 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.508859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c7b58e-7495-48f7-8716-999531e4e015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg\" (UID: \"85c7b58e-7495-48f7-8716-999531e4e015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"
Jul 17 19:28:24.509528 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.509509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c7b58e-7495-48f7-8716-999531e4e015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg\" (UID: \"85c7b58e-7495-48f7-8716-999531e4e015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"
Jul 17 19:28:24.631598 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.631566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:28:24.692807 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.692767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"
Jul 17 19:28:24.824102 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.824077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"]
Jul 17 19:28:24.826620 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:28:24.826580    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c7b58e_7495_48f7_8716_999531e4e015.slice/crio-5e18be9b6c3c9820c7cdd12e490c58a6562e21974e3383fad6b3cc07df0dbf9a WatchSource:0}: Error finding container 5e18be9b6c3c9820c7cdd12e490c58a6562e21974e3383fad6b3cc07df0dbf9a: Status 404 returned error can't find the container with id 5e18be9b6c3c9820c7cdd12e490c58a6562e21974e3383fad6b3cc07df0dbf9a
Jul 17 19:28:24.840652 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.840621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg" event={"ID":"85c7b58e-7495-48f7-8716-999531e4e015","Type":"ContainerStarted","Data":"5e18be9b6c3c9820c7cdd12e490c58a6562e21974e3383fad6b3cc07df0dbf9a"}
Jul 17 19:28:24.842185 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.842161    2563 generic.go:356] "Generic (PLEG): container finished" podID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerID="505d3fc9bc325086a411fce26e366b31f0d56f8b1f3713af8a600417c0b90e97" exitCode=6
Jul 17 19:28:24.842274 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.842241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml" event={"ID":"9ab789a6-3304-46c8-9d28-ce1b26371bdf","Type":"ContainerDied","Data":"505d3fc9bc325086a411fce26e366b31f0d56f8b1f3713af8a600417c0b90e97"}
Jul 17 19:28:24.842506 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:24.842487    2563 scope.go:117] "RemoveContainer" containerID="505d3fc9bc325086a411fce26e366b31f0d56f8b1f3713af8a600417c0b90e97"
Jul 17 19:28:25.848187 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:25.848155    2563 generic.go:356] "Generic (PLEG): container finished" podID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerID="64b8104606d995a90d9173859d2d6208abe21a150fef9d5b1112e58a272bfe03" exitCode=6
Jul 17 19:28:25.848647 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:25.848289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml" event={"ID":"9ab789a6-3304-46c8-9d28-ce1b26371bdf","Type":"ContainerDied","Data":"64b8104606d995a90d9173859d2d6208abe21a150fef9d5b1112e58a272bfe03"}
Jul 17 19:28:25.848647 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:25.848345    2563 scope.go:117] "RemoveContainer" containerID="505d3fc9bc325086a411fce26e366b31f0d56f8b1f3713af8a600417c0b90e97"
Jul 17 19:28:25.848647 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:25.848550    2563 scope.go:117] "RemoveContainer" containerID="64b8104606d995a90d9173859d2d6208abe21a150fef9d5b1112e58a272bfe03"
Jul 17 19:28:25.848820 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:28:25.848784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml_odh-ai-gateway-infra(9ab789a6-3304-46c8-9d28-ce1b26371bdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf"
Jul 17 19:28:25.849970 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:25.849941    2563 generic.go:356] "Generic (PLEG): container finished" podID="85c7b58e-7495-48f7-8716-999531e4e015" containerID="8c85100ead46179a1aec81812b697ddd13fc6960f9db28830854ed2eb2006bac" exitCode=6
Jul 17 19:28:25.850058 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:25.849970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg" event={"ID":"85c7b58e-7495-48f7-8716-999531e4e015","Type":"ContainerDied","Data":"8c85100ead46179a1aec81812b697ddd13fc6960f9db28830854ed2eb2006bac"}
Jul 17 19:28:25.850182 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:25.850168    2563 scope.go:117] "RemoveContainer" containerID="8c85100ead46179a1aec81812b697ddd13fc6960f9db28830854ed2eb2006bac"
Jul 17 19:28:26.855915 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:26.855885    2563 scope.go:117] "RemoveContainer" containerID="64b8104606d995a90d9173859d2d6208abe21a150fef9d5b1112e58a272bfe03"
Jul 17 19:28:26.856386 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:28:26.856092    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml_odh-ai-gateway-infra(9ab789a6-3304-46c8-9d28-ce1b26371bdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf"
Jul 17 19:28:26.857537 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:26.857512    2563 generic.go:356] "Generic (PLEG): container finished" podID="85c7b58e-7495-48f7-8716-999531e4e015" containerID="3c6f34f6425e33c14f9d2ff7625e00206d1a973fd5a32dc4a0593f83fa2731c2" exitCode=6
Jul 17 19:28:26.857648 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:26.857545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg" event={"ID":"85c7b58e-7495-48f7-8716-999531e4e015","Type":"ContainerDied","Data":"3c6f34f6425e33c14f9d2ff7625e00206d1a973fd5a32dc4a0593f83fa2731c2"}
Jul 17 19:28:26.857648 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:26.857588    2563 scope.go:117] "RemoveContainer" containerID="8c85100ead46179a1aec81812b697ddd13fc6960f9db28830854ed2eb2006bac"
Jul 17 19:28:26.857774 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:26.857759    2563 scope.go:117] "RemoveContainer" containerID="3c6f34f6425e33c14f9d2ff7625e00206d1a973fd5a32dc4a0593f83fa2731c2"
Jul 17 19:28:26.857975 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:28:26.857953    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg_odh-ai-gateway-infra(85c7b58e-7495-48f7-8716-999531e4e015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg" podUID="85c7b58e-7495-48f7-8716-999531e4e015"
Jul 17 19:28:27.869691 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:27.869661    2563 scope.go:117] "RemoveContainer" containerID="3c6f34f6425e33c14f9d2ff7625e00206d1a973fd5a32dc4a0593f83fa2731c2"
Jul 17 19:28:27.870153 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:28:27.869845    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg_odh-ai-gateway-infra(85c7b58e-7495-48f7-8716-999531e4e015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg" podUID="85c7b58e-7495-48f7-8716-999531e4e015"
Jul 17 19:28:38.689528 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:38.689444    2563 scope.go:117] "RemoveContainer" containerID="3c6f34f6425e33c14f9d2ff7625e00206d1a973fd5a32dc4a0593f83fa2731c2"
Jul 17 19:28:39.922793 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:39.922762    2563 generic.go:356] "Generic (PLEG): container finished" podID="85c7b58e-7495-48f7-8716-999531e4e015" containerID="4bf5422e1b1c86cef51feffc5e987df5d52a544d4c6522a41c4bffed130a22a0" exitCode=6
Jul 17 19:28:39.922793 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:39.922785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg" event={"ID":"85c7b58e-7495-48f7-8716-999531e4e015","Type":"ContainerDied","Data":"4bf5422e1b1c86cef51feffc5e987df5d52a544d4c6522a41c4bffed130a22a0"}
Jul 17 19:28:39.923309 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:39.922856    2563 scope.go:117] "RemoveContainer" containerID="3c6f34f6425e33c14f9d2ff7625e00206d1a973fd5a32dc4a0593f83fa2731c2"
Jul 17 19:28:39.923309 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:39.923131    2563 scope.go:117] "RemoveContainer" containerID="4bf5422e1b1c86cef51feffc5e987df5d52a544d4c6522a41c4bffed130a22a0"
Jul 17 19:28:39.923391 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:28:39.923360    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg_odh-ai-gateway-infra(85c7b58e-7495-48f7-8716-999531e4e015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg" podUID="85c7b58e-7495-48f7-8716-999531e4e015"
Jul 17 19:28:40.689736 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:40.689697    2563 scope.go:117] "RemoveContainer" containerID="64b8104606d995a90d9173859d2d6208abe21a150fef9d5b1112e58a272bfe03"
Jul 17 19:28:40.948172 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:40.948071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"]
Jul 17 19:28:41.081013 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.080986    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"
Jul 17 19:28:41.157642 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.157608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c7b58e-7495-48f7-8716-999531e4e015-maas-api-service-ca\") pod \"85c7b58e-7495-48f7-8716-999531e4e015\" (UID: \"85c7b58e-7495-48f7-8716-999531e4e015\") "
Jul 17 19:28:41.157965 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.157943    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c7b58e-7495-48f7-8716-999531e4e015-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85c7b58e-7495-48f7-8716-999531e4e015" (UID: "85c7b58e-7495-48f7-8716-999531e4e015"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:28:41.259086 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.259002    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c7b58e-7495-48f7-8716-999531e4e015-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:28:41.298969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.298937    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:28:41.934786 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.934749    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"
Jul 17 19:28:41.934984 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.934757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg" event={"ID":"85c7b58e-7495-48f7-8716-999531e4e015","Type":"ContainerDied","Data":"5e18be9b6c3c9820c7cdd12e490c58a6562e21974e3383fad6b3cc07df0dbf9a"}
Jul 17 19:28:41.934984 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.934879    2563 scope.go:117] "RemoveContainer" containerID="4bf5422e1b1c86cef51feffc5e987df5d52a544d4c6522a41c4bffed130a22a0"
Jul 17 19:28:41.936932 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.936890    2563 generic.go:356] "Generic (PLEG): container finished" podID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerID="cf7c274bbbe07300a2cf480c6656855af078f0bbc87835d0abb572821df8e50d" exitCode=6
Jul 17 19:28:41.937035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.936956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml" event={"ID":"9ab789a6-3304-46c8-9d28-ce1b26371bdf","Type":"ContainerDied","Data":"cf7c274bbbe07300a2cf480c6656855af078f0bbc87835d0abb572821df8e50d"}
Jul 17 19:28:41.937348 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.937331    2563 scope.go:117] "RemoveContainer" containerID="cf7c274bbbe07300a2cf480c6656855af078f0bbc87835d0abb572821df8e50d"
Jul 17 19:28:41.937545 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:28:41.937527    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml_odh-ai-gateway-infra(9ab789a6-3304-46c8-9d28-ce1b26371bdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf"
Jul 17 19:28:41.943038 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.943018    2563 scope.go:117] "RemoveContainer" containerID="64b8104606d995a90d9173859d2d6208abe21a150fef9d5b1112e58a272bfe03"
Jul 17 19:28:41.950903 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.950878    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"]
Jul 17 19:28:41.953203 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:41.953181    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r5wjg"]
Jul 17 19:28:42.981850 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:42.981812    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"]
Jul 17 19:28:43.112498 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.112466    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"
Jul 17 19:28:43.174897 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.174864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab789a6-3304-46c8-9d28-ce1b26371bdf-maas-api-service-ca\") pod \"9ab789a6-3304-46c8-9d28-ce1b26371bdf\" (UID: \"9ab789a6-3304-46c8-9d28-ce1b26371bdf\") "
Jul 17 19:28:43.175281 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.175261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab789a6-3304-46c8-9d28-ce1b26371bdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ab789a6-3304-46c8-9d28-ce1b26371bdf" (UID: "9ab789a6-3304-46c8-9d28-ce1b26371bdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:28:43.276387 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.276300    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab789a6-3304-46c8-9d28-ce1b26371bdf-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:28:43.694622 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.694585    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85c7b58e-7495-48f7-8716-999531e4e015" path="/var/lib/kubelet/pods/85c7b58e-7495-48f7-8716-999531e4e015/volumes"
Jul 17 19:28:43.948030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.947934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml" event={"ID":"9ab789a6-3304-46c8-9d28-ce1b26371bdf","Type":"ContainerDied","Data":"c859c301c831b376f3ab049fe142c4fdacc8eb0baa61eb83aa02833ec3e8eb83"}
Jul 17 19:28:43.948030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.947957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"
Jul 17 19:28:43.948030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.947997    2563 scope.go:117] "RemoveContainer" containerID="cf7c274bbbe07300a2cf480c6656855af078f0bbc87835d0abb572821df8e50d"
Jul 17 19:28:43.966274 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.966249    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"]
Jul 17 19:28:43.970261 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:43.970236    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-pzdml"]
Jul 17 19:28:45.694569 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:45.694535    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" path="/var/lib/kubelet/pods/9ab789a6-3304-46c8-9d28-ce1b26371bdf/volumes"
Jul 17 19:28:58.899192 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:28:58.899151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:29:14.419557 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419514    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"]
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419824    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419835    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419843    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419848    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419861    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419867    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419875    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419886    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419891    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419900    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419904    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419951    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419959    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419964    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419969    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.420027 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.419974    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab789a6-3304-46c8-9d28-ce1b26371bdf" containerName="revoke-keys"
Jul 17 19:29:14.424488 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.424458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"
Jul 17 19:29:14.426803 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.426784    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:29:14.429889 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.429863    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"]
Jul 17 19:29:14.447079 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.447048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be942a-6f91-4bc2-ab8b-7a8320765c22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx\" (UID: \"f4be942a-6f91-4bc2-ab8b-7a8320765c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"
Jul 17 19:29:14.548430 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.548385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be942a-6f91-4bc2-ab8b-7a8320765c22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx\" (UID: \"f4be942a-6f91-4bc2-ab8b-7a8320765c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"
Jul 17 19:29:14.549042 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.549024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be942a-6f91-4bc2-ab8b-7a8320765c22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx\" (UID: \"f4be942a-6f91-4bc2-ab8b-7a8320765c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"
Jul 17 19:29:14.700077 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.699992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:29:14.735464 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.735433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"
Jul 17 19:29:14.868172 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:14.868039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"]
Jul 17 19:29:14.870849 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:29:14.870821    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4be942a_6f91_4bc2_ab8b_7a8320765c22.slice/crio-f88050f36d1cfda55e50c7871eaa796b19b4d642d97ba3bba74f3e45ff6cd187 WatchSource:0}: Error finding container f88050f36d1cfda55e50c7871eaa796b19b4d642d97ba3bba74f3e45ff6cd187: Status 404 returned error can't find the container with id f88050f36d1cfda55e50c7871eaa796b19b4d642d97ba3bba74f3e45ff6cd187
Jul 17 19:29:15.083094 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:15.083015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" event={"ID":"f4be942a-6f91-4bc2-ab8b-7a8320765c22","Type":"ContainerStarted","Data":"f88050f36d1cfda55e50c7871eaa796b19b4d642d97ba3bba74f3e45ff6cd187"}
Jul 17 19:29:16.088183 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:16.088113    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerID="e8c533585683af6baa7008d2c629ebca6b7cc8d7ab1fa94e1bb30b88310ba696" exitCode=6
Jul 17 19:29:16.088183 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:16.088147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" event={"ID":"f4be942a-6f91-4bc2-ab8b-7a8320765c22","Type":"ContainerDied","Data":"e8c533585683af6baa7008d2c629ebca6b7cc8d7ab1fa94e1bb30b88310ba696"}
Jul 17 19:29:16.088760 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:16.088417    2563 scope.go:117] "RemoveContainer" containerID="e8c533585683af6baa7008d2c629ebca6b7cc8d7ab1fa94e1bb30b88310ba696"
Jul 17 19:29:17.094843 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:17.094810    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerID="2537d99f9b687e74d75a7f81dfae9fe51277b7558aec02ce310e7b3faba32a51" exitCode=6
Jul 17 19:29:17.095344 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:17.094867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" event={"ID":"f4be942a-6f91-4bc2-ab8b-7a8320765c22","Type":"ContainerDied","Data":"2537d99f9b687e74d75a7f81dfae9fe51277b7558aec02ce310e7b3faba32a51"}
Jul 17 19:29:17.095344 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:17.094934    2563 scope.go:117] "RemoveContainer" containerID="e8c533585683af6baa7008d2c629ebca6b7cc8d7ab1fa94e1bb30b88310ba696"
Jul 17 19:29:17.095344 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:17.095170    2563 scope.go:117] "RemoveContainer" containerID="2537d99f9b687e74d75a7f81dfae9fe51277b7558aec02ce310e7b3faba32a51"
Jul 17 19:29:17.095508 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:29:17.095450    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx_odh-ai-gateway-infra(f4be942a-6f91-4bc2-ab8b-7a8320765c22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22"
Jul 17 19:29:18.102032 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:18.102001    2563 scope.go:117] "RemoveContainer" containerID="2537d99f9b687e74d75a7f81dfae9fe51277b7558aec02ce310e7b3faba32a51"
Jul 17 19:29:18.102442 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:29:18.102219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx_odh-ai-gateway-infra(f4be942a-6f91-4bc2-ab8b-7a8320765c22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22"
Jul 17 19:29:30.689259 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:30.689222    2563 scope.go:117] "RemoveContainer" containerID="2537d99f9b687e74d75a7f81dfae9fe51277b7558aec02ce310e7b3faba32a51"
Jul 17 19:29:31.158955 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:31.158919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" event={"ID":"f4be942a-6f91-4bc2-ab8b-7a8320765c22","Type":"ContainerStarted","Data":"b1305f923425f2ab19d8846834f0ae46b1db04d126395c09c86e0278ba1a4524"}
Jul 17 19:29:31.175498 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:31.175443    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" podStartSLOduration=17.175428436 podStartE2EDuration="17.175428436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:29:31.174002874 +0000 UTC m=+3380.084829638" watchObservedRunningTime="2026-07-17 19:29:31.175428436 +0000 UTC m=+3380.086255197"
Jul 17 19:29:31.946351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:31.946310    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:29:32.164699 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:32.164661    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerID="b1305f923425f2ab19d8846834f0ae46b1db04d126395c09c86e0278ba1a4524" exitCode=6
Jul 17 19:29:32.164877 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:32.164709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" event={"ID":"f4be942a-6f91-4bc2-ab8b-7a8320765c22","Type":"ContainerDied","Data":"b1305f923425f2ab19d8846834f0ae46b1db04d126395c09c86e0278ba1a4524"}
Jul 17 19:29:32.164877 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:32.164757    2563 scope.go:117] "RemoveContainer" containerID="2537d99f9b687e74d75a7f81dfae9fe51277b7558aec02ce310e7b3faba32a51"
Jul 17 19:29:32.165057 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:32.165034    2563 scope.go:117] "RemoveContainer" containerID="b1305f923425f2ab19d8846834f0ae46b1db04d126395c09c86e0278ba1a4524"
Jul 17 19:29:32.165310 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:29:32.165287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx_odh-ai-gateway-infra(f4be942a-6f91-4bc2-ab8b-7a8320765c22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22"
Jul 17 19:29:32.204564 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:32.204460    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"]
Jul 17 19:29:33.299080 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:33.299052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"
Jul 17 19:29:33.417170 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:33.417107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be942a-6f91-4bc2-ab8b-7a8320765c22-maas-api-service-ca\") pod \"f4be942a-6f91-4bc2-ab8b-7a8320765c22\" (UID: \"f4be942a-6f91-4bc2-ab8b-7a8320765c22\") "
Jul 17 19:29:33.417519 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:33.417489    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4be942a-6f91-4bc2-ab8b-7a8320765c22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4be942a-6f91-4bc2-ab8b-7a8320765c22" (UID: "f4be942a-6f91-4bc2-ab8b-7a8320765c22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:29:33.518532 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:33.518442    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4be942a-6f91-4bc2-ab8b-7a8320765c22-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:29:34.176644 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:34.176602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx" event={"ID":"f4be942a-6f91-4bc2-ab8b-7a8320765c22","Type":"ContainerDied","Data":"f88050f36d1cfda55e50c7871eaa796b19b4d642d97ba3bba74f3e45ff6cd187"}
Jul 17 19:29:34.176829 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:34.176679    2563 scope.go:117] "RemoveContainer" containerID="b1305f923425f2ab19d8846834f0ae46b1db04d126395c09c86e0278ba1a4524"
Jul 17 19:29:34.176829 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:34.176686    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"
Jul 17 19:29:34.193083 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:34.193054    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"]
Jul 17 19:29:34.194281 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:34.194256    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-8wwqx"]
Jul 17 19:29:35.694263 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:35.694230    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" path="/var/lib/kubelet/pods/f4be942a-6f91-4bc2-ab8b-7a8320765c22/volumes"
Jul 17 19:29:41.563089 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.563046    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"]
Jul 17 19:29:41.563609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.563529    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:29:41.563609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.563548    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:29:41.563609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.563581    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:29:41.563609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.563588    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:29:41.563819 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.563652    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:29:41.563819 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.563665    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:29:41.563819 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.563678    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c7b58e-7495-48f7-8716-999531e4e015" containerName="revoke-keys"
Jul 17 19:29:41.568032 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.568009    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"
Jul 17 19:29:41.570547 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.570526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:29:41.574743 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.574700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"]
Jul 17 19:29:41.691431 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.691397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac18f4f2-3584-476e-9989-c15707b1cd02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9\" (UID: \"ac18f4f2-3584-476e-9989-c15707b1cd02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"
Jul 17 19:29:41.792173 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.792110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac18f4f2-3584-476e-9989-c15707b1cd02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9\" (UID: \"ac18f4f2-3584-476e-9989-c15707b1cd02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"
Jul 17 19:29:41.792946 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.792917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac18f4f2-3584-476e-9989-c15707b1cd02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9\" (UID: \"ac18f4f2-3584-476e-9989-c15707b1cd02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"
Jul 17 19:29:41.879263 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:41.879228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"
Jul 17 19:29:42.008473 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:42.008420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"]
Jul 17 19:29:42.010646 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:29:42.010615    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac18f4f2_3584_476e_9989_c15707b1cd02.slice/crio-9cc29b6280672d046bcb54fc2d17feaba3d709ebdf1140d4ea9a1c59d355623f WatchSource:0}: Error finding container 9cc29b6280672d046bcb54fc2d17feaba3d709ebdf1140d4ea9a1c59d355623f: Status 404 returned error can't find the container with id 9cc29b6280672d046bcb54fc2d17feaba3d709ebdf1140d4ea9a1c59d355623f
Jul 17 19:29:42.211731 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:42.211699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" event={"ID":"ac18f4f2-3584-476e-9989-c15707b1cd02","Type":"ContainerStarted","Data":"84b173c611c23763eb0fdc610c0be0c728c8dbd0e88adbc3cade32cdb2b8f9dd"}
Jul 17 19:29:42.211896 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:42.211757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" event={"ID":"ac18f4f2-3584-476e-9989-c15707b1cd02","Type":"ContainerStarted","Data":"9cc29b6280672d046bcb54fc2d17feaba3d709ebdf1140d4ea9a1c59d355623f"}
Jul 17 19:29:42.227773 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:42.227726    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" podStartSLOduration=1.227709406 podStartE2EDuration="1.227709406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:29:42.227270572 +0000 UTC m=+3391.138097338" watchObservedRunningTime="2026-07-17 19:29:42.227709406 +0000 UTC m=+3391.138536166"
Jul 17 19:29:43.217397 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:43.217364    2563 generic.go:356] "Generic (PLEG): container finished" podID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerID="84b173c611c23763eb0fdc610c0be0c728c8dbd0e88adbc3cade32cdb2b8f9dd" exitCode=6
Jul 17 19:29:43.217906 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:43.217470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" event={"ID":"ac18f4f2-3584-476e-9989-c15707b1cd02","Type":"ContainerDied","Data":"84b173c611c23763eb0fdc610c0be0c728c8dbd0e88adbc3cade32cdb2b8f9dd"}
Jul 17 19:29:43.217906 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:43.217698    2563 scope.go:117] "RemoveContainer" containerID="84b173c611c23763eb0fdc610c0be0c728c8dbd0e88adbc3cade32cdb2b8f9dd"
Jul 17 19:29:44.223860 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:44.223768    2563 generic.go:356] "Generic (PLEG): container finished" podID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerID="a37b192e27a9257354da76c783f3d04a4a1c2ded9be7f248d7e0ab0106628902" exitCode=6
Jul 17 19:29:44.223860 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:44.223820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" event={"ID":"ac18f4f2-3584-476e-9989-c15707b1cd02","Type":"ContainerDied","Data":"a37b192e27a9257354da76c783f3d04a4a1c2ded9be7f248d7e0ab0106628902"}
Jul 17 19:29:44.224401 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:44.223869    2563 scope.go:117] "RemoveContainer" containerID="84b173c611c23763eb0fdc610c0be0c728c8dbd0e88adbc3cade32cdb2b8f9dd"
Jul 17 19:29:44.224401 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:44.224087    2563 scope.go:117] "RemoveContainer" containerID="a37b192e27a9257354da76c783f3d04a4a1c2ded9be7f248d7e0ab0106628902"
Jul 17 19:29:44.224401 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:29:44.224361    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9_odh-ai-gateway-infra(ac18f4f2-3584-476e-9989-c15707b1cd02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02"
Jul 17 19:29:45.229539 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:45.229511    2563 scope.go:117] "RemoveContainer" containerID="a37b192e27a9257354da76c783f3d04a4a1c2ded9be7f248d7e0ab0106628902"
Jul 17 19:29:45.229946 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:29:45.229689    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9_odh-ai-gateway-infra(ac18f4f2-3584-476e-9989-c15707b1cd02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02"
Jul 17 19:29:45.877745 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:45.877711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:29:46.733209 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:46.733148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:29:50.329995 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:50.329958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:29:51.003396 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:51.003358    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:29:58.689638 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:58.689607    2563 scope.go:117] "RemoveContainer" containerID="a37b192e27a9257354da76c783f3d04a4a1c2ded9be7f248d7e0ab0106628902"
Jul 17 19:29:59.290893 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:59.290863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" event={"ID":"ac18f4f2-3584-476e-9989-c15707b1cd02","Type":"ContainerStarted","Data":"f6cf382dd7dcd1e0e1c9b3c3c898ac924d1c6cb273300c8e1f4cec47690e8d0b"}
Jul 17 19:29:59.291227 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:29:59.291211    2563 scope.go:117] "RemoveContainer" containerID="f6cf382dd7dcd1e0e1c9b3c3c898ac924d1c6cb273300c8e1f4cec47690e8d0b"
Jul 17 19:29:59.291424 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:29:59.291393    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9_odh-ai-gateway-infra(ac18f4f2-3584-476e-9989-c15707b1cd02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02"
Jul 17 19:30:00.135082 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.135035    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"]
Jul 17 19:30:00.135550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.135531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:30:00.135595 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.135555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:30:00.135661 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.135648    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4be942a-6f91-4bc2-ab8b-7a8320765c22" containerName="revoke-keys"
Jul 17 19:30:00.138829 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.138812    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"]
Jul 17 19:30:00.138957 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.138939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"
Jul 17 19:30:00.141798 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.141780    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp"]
Jul 17 19:30:00.141925 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.141897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"
Jul 17 19:30:00.144918 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.144809    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp"]
Jul 17 19:30:00.144918 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.144903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp"
Jul 17 19:30:00.149021 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.148986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"]
Jul 17 19:30:00.149994 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.149970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"]
Jul 17 19:30:00.152555 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.152417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"
Jul 17 19:30:00.162532 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.162510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"
Jul 17 19:30:00.194426 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.194400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp"
Jul 17 19:30:00.311156 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.310994    2563 generic.go:356] "Generic (PLEG): container finished" podID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerID="f6cf382dd7dcd1e0e1c9b3c3c898ac924d1c6cb273300c8e1f4cec47690e8d0b" exitCode=6
Jul 17 19:30:00.311156 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.311055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" event={"ID":"ac18f4f2-3584-476e-9989-c15707b1cd02","Type":"ContainerDied","Data":"f6cf382dd7dcd1e0e1c9b3c3c898ac924d1c6cb273300c8e1f4cec47690e8d0b"}
Jul 17 19:30:00.311156 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.311112    2563 scope.go:117] "RemoveContainer" containerID="a37b192e27a9257354da76c783f3d04a4a1c2ded9be7f248d7e0ab0106628902"
Jul 17 19:30:00.311765 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.311605    2563 scope.go:117] "RemoveContainer" containerID="f6cf382dd7dcd1e0e1c9b3c3c898ac924d1c6cb273300c8e1f4cec47690e8d0b"
Jul 17 19:30:00.311894 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:30:00.311807    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9_odh-ai-gateway-infra(ac18f4f2-3584-476e-9989-c15707b1cd02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02"
Jul 17 19:30:00.319405 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.319353    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"]
Jul 17 19:30:00.341453 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.341384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"]
Jul 17 19:30:00.344443 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:30:00.344401    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c82c1b9_9fd8_4bce_b585_c17af79f3160.slice/crio-00c1d0c66a9e75a95c497e93d9a51b214e618a8a3c82cc8a398b60f6b5d6a726 WatchSource:0}: Error finding container 00c1d0c66a9e75a95c497e93d9a51b214e618a8a3c82cc8a398b60f6b5d6a726: Status 404 returned error can't find the container with id 00c1d0c66a9e75a95c497e93d9a51b214e618a8a3c82cc8a398b60f6b5d6a726
Jul 17 19:30:00.364382 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.364352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"]
Jul 17 19:30:00.367085 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:30:00.367050    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99847885_baea_483f_80c8_76803a694648.slice/crio-4b7cddd57345bc5a2d43ef6d02471895664688bb7c092a8f5a40461b6691809d WatchSource:0}: Error finding container 4b7cddd57345bc5a2d43ef6d02471895664688bb7c092a8f5a40461b6691809d: Status 404 returned error can't find the container with id 4b7cddd57345bc5a2d43ef6d02471895664688bb7c092a8f5a40461b6691809d
Jul 17 19:30:00.382440 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:00.382412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp"]
Jul 17 19:30:00.385368 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:30:00.385304    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790a7429_f6bc_4e60_9df7_40d09a00e647.slice/crio-f25abc63e4c7d053361da97878fc5ce55955595d67342ccdaf470e47768fb197 WatchSource:0}: Error finding container f25abc63e4c7d053361da97878fc5ce55955595d67342ccdaf470e47768fb197: Status 404 returned error can't find the container with id f25abc63e4c7d053361da97878fc5ce55955595d67342ccdaf470e47768fb197
Jul 17 19:30:01.316841 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.316807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp" event={"ID":"790a7429-f6bc-4e60-9df7-40d09a00e647","Type":"ContainerStarted","Data":"07492c0ea6acf5555d05491f519a348ddccdbba68c3fb9afabdf46a625e91df9"}
Jul 17 19:30:01.317297 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.316861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp" event={"ID":"790a7429-f6bc-4e60-9df7-40d09a00e647","Type":"ContainerStarted","Data":"f25abc63e4c7d053361da97878fc5ce55955595d67342ccdaf470e47768fb197"}
Jul 17 19:30:01.318475 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.318445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d" event={"ID":"0c82c1b9-9fd8-4bce-b585-c17af79f3160","Type":"ContainerStarted","Data":"d421290e4f91608dd897e014d15fd7f7610e2219b4203a965c438477b8f996a9"}
Jul 17 19:30:01.318619 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.318501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d" event={"ID":"0c82c1b9-9fd8-4bce-b585-c17af79f3160","Type":"ContainerStarted","Data":"00c1d0c66a9e75a95c497e93d9a51b214e618a8a3c82cc8a398b60f6b5d6a726"}
Jul 17 19:30:01.321774 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.321748    2563 generic.go:356] "Generic (PLEG): container finished" podID="99847885-baea-483f-80c8-76803a694648" containerID="6fa2b656416e53b0b30b6e7c646f5e599181f2d50dd16e842198d44921d5d67b" exitCode=0
Jul 17 19:30:01.321910 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.321817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl" event={"ID":"99847885-baea-483f-80c8-76803a694648","Type":"ContainerDied","Data":"6fa2b656416e53b0b30b6e7c646f5e599181f2d50dd16e842198d44921d5d67b"}
Jul 17 19:30:01.321910 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.321852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl" event={"ID":"99847885-baea-483f-80c8-76803a694648","Type":"ContainerStarted","Data":"4b7cddd57345bc5a2d43ef6d02471895664688bb7c092a8f5a40461b6691809d"}
Jul 17 19:30:01.332035 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.331987    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp" podStartSLOduration=1.3319716449999999 podStartE2EDuration="1.331971645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:01.330573286 +0000 UTC m=+3410.241400063" watchObservedRunningTime="2026-07-17 19:30:01.331971645 +0000 UTC m=+3410.242798470"
Jul 17 19:30:01.345696 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.345647    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d" podStartSLOduration=1.345629621 podStartE2EDuration="1.345629621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:01.343979531 +0000 UTC m=+3410.254806293" watchObservedRunningTime="2026-07-17 19:30:01.345629621 +0000 UTC m=+3410.256456382"
Jul 17 19:30:01.454569 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.454546    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"
Jul 17 19:30:01.562806 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.562690    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac18f4f2-3584-476e-9989-c15707b1cd02-maas-api-service-ca\") pod \"ac18f4f2-3584-476e-9989-c15707b1cd02\" (UID: \"ac18f4f2-3584-476e-9989-c15707b1cd02\") "
Jul 17 19:30:01.563081 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.563057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac18f4f2-3584-476e-9989-c15707b1cd02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac18f4f2-3584-476e-9989-c15707b1cd02" (UID: "ac18f4f2-3584-476e-9989-c15707b1cd02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:30:01.663544 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:01.663512    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac18f4f2-3584-476e-9989-c15707b1cd02-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:30:02.076610 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.076575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:30:02.169249 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.169216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:30:02.327638 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.327543    2563 generic.go:356] "Generic (PLEG): container finished" podID="790a7429-f6bc-4e60-9df7-40d09a00e647" containerID="07492c0ea6acf5555d05491f519a348ddccdbba68c3fb9afabdf46a625e91df9" exitCode=0
Jul 17 19:30:02.327638 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.327602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp" event={"ID":"790a7429-f6bc-4e60-9df7-40d09a00e647","Type":"ContainerDied","Data":"07492c0ea6acf5555d05491f519a348ddccdbba68c3fb9afabdf46a625e91df9"}
Jul 17 19:30:02.329174 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.329147    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c82c1b9-9fd8-4bce-b585-c17af79f3160" containerID="d421290e4f91608dd897e014d15fd7f7610e2219b4203a965c438477b8f996a9" exitCode=0
Jul 17 19:30:02.329308 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.329251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d" event={"ID":"0c82c1b9-9fd8-4bce-b585-c17af79f3160","Type":"ContainerDied","Data":"d421290e4f91608dd897e014d15fd7f7610e2219b4203a965c438477b8f996a9"}
Jul 17 19:30:02.330417 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.330399    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"
Jul 17 19:30:02.330522 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.330431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9" event={"ID":"ac18f4f2-3584-476e-9989-c15707b1cd02","Type":"ContainerDied","Data":"9cc29b6280672d046bcb54fc2d17feaba3d709ebdf1140d4ea9a1c59d355623f"}
Jul 17 19:30:02.330522 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.330478    2563 scope.go:117] "RemoveContainer" containerID="f6cf382dd7dcd1e0e1c9b3c3c898ac924d1c6cb273300c8e1f4cec47690e8d0b"
Jul 17 19:30:02.371652 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.371617    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"]
Jul 17 19:30:02.375372 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.375347    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-p8ft9"]
Jul 17 19:30:02.466651 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:02.466628    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"
Jul 17 19:30:03.336940 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:03.336903    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"
Jul 17 19:30:03.336940 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:03.336927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl" event={"ID":"99847885-baea-483f-80c8-76803a694648","Type":"ContainerDied","Data":"4b7cddd57345bc5a2d43ef6d02471895664688bb7c092a8f5a40461b6691809d"}
Jul 17 19:30:03.337407 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:03.336998    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b7cddd57345bc5a2d43ef6d02471895664688bb7c092a8f5a40461b6691809d"
Jul 17 19:30:03.472036 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:03.472010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp"
Jul 17 19:30:03.505913 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:03.505888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"
Jul 17 19:30:03.694471 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:03.694435    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" path="/var/lib/kubelet/pods/ac18f4f2-3584-476e-9989-c15707b1cd02/volumes"
Jul 17 19:30:04.341988 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:04.341959    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp"
Jul 17 19:30:04.342474 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:04.341961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-b9pqp" event={"ID":"790a7429-f6bc-4e60-9df7-40d09a00e647","Type":"ContainerDied","Data":"f25abc63e4c7d053361da97878fc5ce55955595d67342ccdaf470e47768fb197"}
Jul 17 19:30:04.342474 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:04.342095    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f25abc63e4c7d053361da97878fc5ce55955595d67342ccdaf470e47768fb197"
Jul 17 19:30:04.343476 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:04.343454    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"
Jul 17 19:30:04.343618 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:04.343483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d" event={"ID":"0c82c1b9-9fd8-4bce-b585-c17af79f3160","Type":"ContainerDied","Data":"00c1d0c66a9e75a95c497e93d9a51b214e618a8a3c82cc8a398b60f6b5d6a726"}
Jul 17 19:30:04.343618 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:04.343516    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00c1d0c66a9e75a95c497e93d9a51b214e618a8a3c82cc8a398b60f6b5d6a726"
Jul 17 19:30:06.201373 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:06.201336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:30:24.737299 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:24.737253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:30:29.398211 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:29.398168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:30:42.494013 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:42.493972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:30:42.596033 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:42.595996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:30:44.202454 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:44.202422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:30:54.488735 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.488699    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"]
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.488998    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489016    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489023    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489031    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c82c1b9-9fd8-4bce-b585-c17af79f3160" containerName="cleanup"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489036    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c82c1b9-9fd8-4bce-b585-c17af79f3160" containerName="cleanup"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489043    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99847885-baea-483f-80c8-76803a694648" containerName="cleanup"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489048    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99847885-baea-483f-80c8-76803a694648" containerName="cleanup"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489058    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489063    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489073    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790a7429-f6bc-4e60-9df7-40d09a00e647" containerName="cleanup"
Jul 17 19:30:54.489113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489078    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="790a7429-f6bc-4e60-9df7-40d09a00e647" containerName="cleanup"
Jul 17 19:30:54.489520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489169    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99847885-baea-483f-80c8-76803a694648" containerName="cleanup"
Jul 17 19:30:54.489520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489178    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="790a7429-f6bc-4e60-9df7-40d09a00e647" containerName="cleanup"
Jul 17 19:30:54.489520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489183    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c82c1b9-9fd8-4bce-b585-c17af79f3160" containerName="cleanup"
Jul 17 19:30:54.489520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489190    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:30:54.489520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.489195    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:30:54.492103 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.492085    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"
Jul 17 19:30:54.494594 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.494570    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:30:54.499505 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.498803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"]
Jul 17 19:30:54.513237 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.513204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68cbe5-64e5-4cd5-97d2-bfbc807cb003-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx\" (UID: \"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"
Jul 17 19:30:54.614357 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.614314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68cbe5-64e5-4cd5-97d2-bfbc807cb003-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx\" (UID: \"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"
Jul 17 19:30:54.614952 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.614930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68cbe5-64e5-4cd5-97d2-bfbc807cb003-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx\" (UID: \"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"
Jul 17 19:30:54.803155 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.803036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"
Jul 17 19:30:54.940506 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:54.940451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"]
Jul 17 19:30:54.944514 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:30:54.944477    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d68cbe5_64e5_4cd5_97d2_bfbc807cb003.slice/crio-98619aaddcd82f43bb9ac25e2f57536a6ae7d07e65a60caf1be0fce00cfd58b1 WatchSource:0}: Error finding container 98619aaddcd82f43bb9ac25e2f57536a6ae7d07e65a60caf1be0fce00cfd58b1: Status 404 returned error can't find the container with id 98619aaddcd82f43bb9ac25e2f57536a6ae7d07e65a60caf1be0fce00cfd58b1
Jul 17 19:30:55.575992 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:55.575952    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" event={"ID":"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003","Type":"ContainerStarted","Data":"cb47e5edfef904a73f28b0811ab2146cc1b9c57fcda93d74bb650b6de7aa92af"}
Jul 17 19:30:55.576435 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:55.576059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" event={"ID":"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003","Type":"ContainerStarted","Data":"98619aaddcd82f43bb9ac25e2f57536a6ae7d07e65a60caf1be0fce00cfd58b1"}
Jul 17 19:30:55.592778 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:55.592721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" podStartSLOduration=1.59270281 podStartE2EDuration="1.59270281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:55.592107017 +0000 UTC m=+3464.502933779" watchObservedRunningTime="2026-07-17 19:30:55.59270281 +0000 UTC m=+3464.503529571"
Jul 17 19:30:56.581830 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:56.581799    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerID="cb47e5edfef904a73f28b0811ab2146cc1b9c57fcda93d74bb650b6de7aa92af" exitCode=6
Jul 17 19:30:56.582262 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:56.581892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" event={"ID":"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003","Type":"ContainerDied","Data":"cb47e5edfef904a73f28b0811ab2146cc1b9c57fcda93d74bb650b6de7aa92af"}
Jul 17 19:30:56.582262 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:56.582023    2563 scope.go:117] "RemoveContainer" containerID="cb47e5edfef904a73f28b0811ab2146cc1b9c57fcda93d74bb650b6de7aa92af"
Jul 17 19:30:57.588787 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:57.588702    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerID="906f6c95152807b0eac6e1c339a42b947f3787c89836259f181889040913cc9b" exitCode=6
Jul 17 19:30:57.588787 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:57.588758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" event={"ID":"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003","Type":"ContainerDied","Data":"906f6c95152807b0eac6e1c339a42b947f3787c89836259f181889040913cc9b"}
Jul 17 19:30:57.589244 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:57.588815    2563 scope.go:117] "RemoveContainer" containerID="cb47e5edfef904a73f28b0811ab2146cc1b9c57fcda93d74bb650b6de7aa92af"
Jul 17 19:30:57.589244 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:57.589012    2563 scope.go:117] "RemoveContainer" containerID="906f6c95152807b0eac6e1c339a42b947f3787c89836259f181889040913cc9b"
Jul 17 19:30:57.589335 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:30:57.589257    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx_odh-ai-gateway-infra(1d68cbe5-64e5-4cd5-97d2-bfbc807cb003)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003"
Jul 17 19:30:58.595336 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:30:58.595304    2563 scope.go:117] "RemoveContainer" containerID="906f6c95152807b0eac6e1c339a42b947f3787c89836259f181889040913cc9b"
Jul 17 19:30:58.595807 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:30:58.595550    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx_odh-ai-gateway-infra(1d68cbe5-64e5-4cd5-97d2-bfbc807cb003)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003"
Jul 17 19:31:02.281377 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:02.281340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:31:03.140822 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:03.140771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:31:07.598878 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:07.598843    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:31:10.689214 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:10.689176    2563 scope.go:117] "RemoveContainer" containerID="906f6c95152807b0eac6e1c339a42b947f3787c89836259f181889040913cc9b"
Jul 17 19:31:11.653540 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:11.653448    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerID="8b1f9cf6d6662ccb9b9ac2712a7cf56256499f4acdb0cdc2813c698ca9209557" exitCode=6
Jul 17 19:31:11.653540 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:11.653517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" event={"ID":"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003","Type":"ContainerDied","Data":"8b1f9cf6d6662ccb9b9ac2712a7cf56256499f4acdb0cdc2813c698ca9209557"}
Jul 17 19:31:11.653767 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:11.653589    2563 scope.go:117] "RemoveContainer" containerID="906f6c95152807b0eac6e1c339a42b947f3787c89836259f181889040913cc9b"
Jul 17 19:31:11.653883 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:11.653862    2563 scope.go:117] "RemoveContainer" containerID="8b1f9cf6d6662ccb9b9ac2712a7cf56256499f4acdb0cdc2813c698ca9209557"
Jul 17 19:31:11.654089 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:31:11.654067    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx_odh-ai-gateway-infra(1d68cbe5-64e5-4cd5-97d2-bfbc807cb003)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003"
Jul 17 19:31:12.680228 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:12.680193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"]
Jul 17 19:31:12.809026 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:12.808997    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"
Jul 17 19:31:12.871176 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:12.871136    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68cbe5-64e5-4cd5-97d2-bfbc807cb003-maas-api-service-ca\") pod \"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003\" (UID: \"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003\") "
Jul 17 19:31:12.871482 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:12.871460    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d68cbe5-64e5-4cd5-97d2-bfbc807cb003-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" (UID: "1d68cbe5-64e5-4cd5-97d2-bfbc807cb003"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:31:12.972318 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:12.972206    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d68cbe5-64e5-4cd5-97d2-bfbc807cb003-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:31:13.664018 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:13.663987    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"
Jul 17 19:31:13.664018 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:13.664008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx" event={"ID":"1d68cbe5-64e5-4cd5-97d2-bfbc807cb003","Type":"ContainerDied","Data":"98619aaddcd82f43bb9ac25e2f57536a6ae7d07e65a60caf1be0fce00cfd58b1"}
Jul 17 19:31:13.664393 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:13.664067    2563 scope.go:117] "RemoveContainer" containerID="8b1f9cf6d6662ccb9b9ac2712a7cf56256499f4acdb0cdc2813c698ca9209557"
Jul 17 19:31:13.684601 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:13.684574    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"]
Jul 17 19:31:13.699808 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:13.699776    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-rdncx"]
Jul 17 19:31:15.693630 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:15.693593    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" path="/var/lib/kubelet/pods/1d68cbe5-64e5-4cd5-97d2-bfbc807cb003/volumes"
Jul 17 19:31:18.634467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:18.634440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:31:22.504651 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:22.504610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:31:28.333707 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:28.333669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:31:41.838443 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:41.838376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:31:46.344212 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344171    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"]
Jul 17 19:31:46.344702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344520    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:31:46.344702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:31:46.344702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344551    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:31:46.344702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344557    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:31:46.344702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344608    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:31:46.344702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344621    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:31:46.344702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344628    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac18f4f2-3584-476e-9989-c15707b1cd02" containerName="revoke-keys"
Jul 17 19:31:46.344702 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.344635    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:31:46.347888 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.347871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"
Jul 17 19:31:46.350504 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.350481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:31:46.355933 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.355907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"]
Jul 17 19:31:46.447597 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.447552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03606466-a539-4213-b88c-619fa94ca713-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8\" (UID: \"03606466-a539-4213-b88c-619fa94ca713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"
Jul 17 19:31:46.548833 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.548695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03606466-a539-4213-b88c-619fa94ca713-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8\" (UID: \"03606466-a539-4213-b88c-619fa94ca713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"
Jul 17 19:31:46.554674 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.549385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03606466-a539-4213-b88c-619fa94ca713-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8\" (UID: \"03606466-a539-4213-b88c-619fa94ca713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"
Jul 17 19:31:46.658894 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.658799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"
Jul 17 19:31:46.793741 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.793668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"]
Jul 17 19:31:46.796525 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:31:46.796479    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03606466_a539_4213_b88c_619fa94ca713.slice/crio-a0aa8b9e96d9fc66fd341701e4c181b3a0dc5e6c48b7b6b5ceb1cc761cead9ee WatchSource:0}: Error finding container a0aa8b9e96d9fc66fd341701e4c181b3a0dc5e6c48b7b6b5ceb1cc761cead9ee: Status 404 returned error can't find the container with id a0aa8b9e96d9fc66fd341701e4c181b3a0dc5e6c48b7b6b5ceb1cc761cead9ee
Jul 17 19:31:46.808969 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.808941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:31:46.811244 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:46.811220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8" event={"ID":"03606466-a539-4213-b88c-619fa94ca713","Type":"ContainerStarted","Data":"a0aa8b9e96d9fc66fd341701e4c181b3a0dc5e6c48b7b6b5ceb1cc761cead9ee"}
Jul 17 19:31:47.772751 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:47.772657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:31:47.817512 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:47.817479    2563 generic.go:356] "Generic (PLEG): container finished" podID="03606466-a539-4213-b88c-619fa94ca713" containerID="12ff85188ddf628496a13b757a83054da987d62f12bf28808e555b88370e0d96" exitCode=6
Jul 17 19:31:47.817685 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:47.817522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8" event={"ID":"03606466-a539-4213-b88c-619fa94ca713","Type":"ContainerDied","Data":"12ff85188ddf628496a13b757a83054da987d62f12bf28808e555b88370e0d96"}
Jul 17 19:31:47.817760 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:47.817747    2563 scope.go:117] "RemoveContainer" containerID="12ff85188ddf628496a13b757a83054da987d62f12bf28808e555b88370e0d96"
Jul 17 19:31:47.837908 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:47.837879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:31:48.823617 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:48.823578    2563 generic.go:356] "Generic (PLEG): container finished" podID="03606466-a539-4213-b88c-619fa94ca713" containerID="5bd37fee6f6c9c33785482215a0e0d403b0e2d1c033f959e3174c3c1d0a002b1" exitCode=6
Jul 17 19:31:48.824030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:48.823669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8" event={"ID":"03606466-a539-4213-b88c-619fa94ca713","Type":"ContainerDied","Data":"5bd37fee6f6c9c33785482215a0e0d403b0e2d1c033f959e3174c3c1d0a002b1"}
Jul 17 19:31:48.824030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:48.823755    2563 scope.go:117] "RemoveContainer" containerID="12ff85188ddf628496a13b757a83054da987d62f12bf28808e555b88370e0d96"
Jul 17 19:31:48.824030 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:48.823832    2563 scope.go:117] "RemoveContainer" containerID="5bd37fee6f6c9c33785482215a0e0d403b0e2d1c033f959e3174c3c1d0a002b1"
Jul 17 19:31:48.824200 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:31:48.824043    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8_odh-ai-gateway-infra(03606466-a539-4213-b88c-619fa94ca713)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8" podUID="03606466-a539-4213-b88c-619fa94ca713"
Jul 17 19:31:49.829725 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:49.829698    2563 scope.go:117] "RemoveContainer" containerID="5bd37fee6f6c9c33785482215a0e0d403b0e2d1c033f959e3174c3c1d0a002b1"
Jul 17 19:31:49.830105 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:31:49.829873    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8_odh-ai-gateway-infra(03606466-a539-4213-b88c-619fa94ca713)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8" podUID="03606466-a539-4213-b88c-619fa94ca713"
Jul 17 19:31:54.747691 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:54.747660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:31:54.847840 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:31:54.847798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:32:00.689894 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:00.689863    2563 scope.go:117] "RemoveContainer" containerID="5bd37fee6f6c9c33785482215a0e0d403b0e2d1c033f959e3174c3c1d0a002b1"
Jul 17 19:32:01.885276 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:01.885241    2563 generic.go:356] "Generic (PLEG): container finished" podID="03606466-a539-4213-b88c-619fa94ca713" containerID="5788a74c06513b3dde5cf0d5bceaf297cb30fc2a2186bfa3b20863185675bfc2" exitCode=6
Jul 17 19:32:01.885720 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:01.885287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8" event={"ID":"03606466-a539-4213-b88c-619fa94ca713","Type":"ContainerDied","Data":"5788a74c06513b3dde5cf0d5bceaf297cb30fc2a2186bfa3b20863185675bfc2"}
Jul 17 19:32:01.885720 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:01.885341    2563 scope.go:117] "RemoveContainer" containerID="5bd37fee6f6c9c33785482215a0e0d403b0e2d1c033f959e3174c3c1d0a002b1"
Jul 17 19:32:01.885720 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:01.885607    2563 scope.go:117] "RemoveContainer" containerID="5788a74c06513b3dde5cf0d5bceaf297cb30fc2a2186bfa3b20863185675bfc2"
Jul 17 19:32:01.885878 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:32:01.885853    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8_odh-ai-gateway-infra(03606466-a539-4213-b88c-619fa94ca713)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8" podUID="03606466-a539-4213-b88c-619fa94ca713"
Jul 17 19:32:02.497492 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:02.497453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:32:02.909217 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:02.909182    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"]
Jul 17 19:32:03.048741 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.048714    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"
Jul 17 19:32:03.093900 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.093871    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03606466-a539-4213-b88c-619fa94ca713-maas-api-service-ca\") pod \"03606466-a539-4213-b88c-619fa94ca713\" (UID: \"03606466-a539-4213-b88c-619fa94ca713\") "
Jul 17 19:32:03.094272 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.094250    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03606466-a539-4213-b88c-619fa94ca713-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03606466-a539-4213-b88c-619fa94ca713" (UID: "03606466-a539-4213-b88c-619fa94ca713"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:32:03.195569 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.195478    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03606466-a539-4213-b88c-619fa94ca713-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:32:03.896137 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.896084    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"
Jul 17 19:32:03.896321 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.896107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8" event={"ID":"03606466-a539-4213-b88c-619fa94ca713","Type":"ContainerDied","Data":"a0aa8b9e96d9fc66fd341701e4c181b3a0dc5e6c48b7b6b5ceb1cc761cead9ee"}
Jul 17 19:32:03.896321 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.896198    2563 scope.go:117] "RemoveContainer" containerID="5788a74c06513b3dde5cf0d5bceaf297cb30fc2a2186bfa3b20863185675bfc2"
Jul 17 19:32:03.927706 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.927674    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"]
Jul 17 19:32:03.937113 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:03.937083    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v2dd8"]
Jul 17 19:32:05.693488 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:05.693454    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03606466-a539-4213-b88c-619fa94ca713" path="/var/lib/kubelet/pods/03606466-a539-4213-b88c-619fa94ca713/volumes"
Jul 17 19:32:21.405163 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:21.405108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:32:25.894682 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:25.894630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:32:28.142395 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:28.142333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:32:28.936256 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:28.936200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:32:36.801219 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:36.801175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:32:46.440725 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:46.440680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:32:46.540199 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:46.540163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:32:56.113760 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.113722    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"]
Jul 17 19:32:56.114437 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.113964    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj" podUID="773daa09-4b25-466b-b70e-9b5d0d6e28e5" containerName="maas-api" containerID="cri-o://20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84" gracePeriod=30
Jul 17 19:32:56.141284 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.141245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:32:56.142133 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.142099    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"]
Jul 17 19:32:56.144744 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.144721    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-98e305-29738610-6prnl"]
Jul 17 19:32:56.568407 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.568381    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:32:56.663542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.663446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/773daa09-4b25-466b-b70e-9b5d0d6e28e5-openshift-service-ca\") pod \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") "
Jul 17 19:32:56.663542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.663494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrz72\" (UniqueName: \"kubernetes.io/projected/773daa09-4b25-466b-b70e-9b5d0d6e28e5-kube-api-access-vrz72\") pod \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") "
Jul 17 19:32:56.663542 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.663526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls\") pod \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\" (UID: \"773daa09-4b25-466b-b70e-9b5d0d6e28e5\") "
Jul 17 19:32:56.663910 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.663884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773daa09-4b25-466b-b70e-9b5d0d6e28e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "773daa09-4b25-466b-b70e-9b5d0d6e28e5" (UID: "773daa09-4b25-466b-b70e-9b5d0d6e28e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:32:56.665825 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.665804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "773daa09-4b25-466b-b70e-9b5d0d6e28e5" (UID: "773daa09-4b25-466b-b70e-9b5d0d6e28e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:32:56.665932 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.665910    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773daa09-4b25-466b-b70e-9b5d0d6e28e5-kube-api-access-vrz72" (OuterVolumeSpecName: "kube-api-access-vrz72") pod "773daa09-4b25-466b-b70e-9b5d0d6e28e5" (UID: "773daa09-4b25-466b-b70e-9b5d0d6e28e5"). InnerVolumeSpecName "kube-api-access-vrz72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:32:56.764800 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.764747    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/773daa09-4b25-466b-b70e-9b5d0d6e28e5-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:32:56.764800 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.764794    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrz72\" (UniqueName: \"kubernetes.io/projected/773daa09-4b25-466b-b70e-9b5d0d6e28e5-kube-api-access-vrz72\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:32:56.764800 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:56.764804    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/773daa09-4b25-466b-b70e-9b5d0d6e28e5-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:32:57.127160 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.127102    2563 generic.go:356] "Generic (PLEG): container finished" podID="773daa09-4b25-466b-b70e-9b5d0d6e28e5" containerID="20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84" exitCode=0
Jul 17 19:32:57.127703 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.127169    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"
Jul 17 19:32:57.127703 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.127189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj" event={"ID":"773daa09-4b25-466b-b70e-9b5d0d6e28e5","Type":"ContainerDied","Data":"20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84"}
Jul 17 19:32:57.127703 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.127246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj" event={"ID":"773daa09-4b25-466b-b70e-9b5d0d6e28e5","Type":"ContainerDied","Data":"f551c234e10d3c7d532a1754a43769556d6008bc020e30ce11ba3da4dbc1fada"}
Jul 17 19:32:57.127703 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.127268    2563 scope.go:117] "RemoveContainer" containerID="20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84"
Jul 17 19:32:57.136226 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.136205    2563 scope.go:117] "RemoveContainer" containerID="20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84"
Jul 17 19:32:57.136535 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:32:57.136515    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84\": container with ID starting with 20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84 not found: ID does not exist" containerID="20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84"
Jul 17 19:32:57.136584 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.136546    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84"} err="failed to get container status \"20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84\": rpc error: code = NotFound desc = could not find container \"20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84\": container with ID starting with 20aeae5d7cefc504f59ab0355d8cb8d2c055f56cdadd05f54713ef0928911c84 not found: ID does not exist"
Jul 17 19:32:57.149867 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.149832    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"]
Jul 17 19:32:57.154176 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.154147    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-98e305-f55859b5b-2q9xj"]
Jul 17 19:32:57.694279 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.694233    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="773daa09-4b25-466b-b70e-9b5d0d6e28e5" path="/var/lib/kubelet/pods/773daa09-4b25-466b-b70e-9b5d0d6e28e5/volumes"
Jul 17 19:32:57.694614 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:57.694595    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99847885-baea-483f-80c8-76803a694648" path="/var/lib/kubelet/pods/99847885-baea-483f-80c8-76803a694648/volumes"
Jul 17 19:32:59.701371 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:32:59.701329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:33:00.818186 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:00.818148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:07.278338 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:07.278219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:07.346517 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:07.346463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:11.474273 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:11.474242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:11.707317 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:11.707284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:11.780950 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:11.780864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:33:11.788830 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:11.788805    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g49xg_eb5d4daf-45c2-42f2-92fc-21c7e1e30083/ovn-acl-logging/0.log"
Jul 17 19:33:12.502516 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:12.502489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:12.590377 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:12.590342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:22.211180 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.211145    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"]
Jul 17 19:33:22.211615 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.211413    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg" podUID="f9292d38-97ce-4073-9841-00123362c521" containerName="maas-api" containerID="cri-o://5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d" gracePeriod=30
Jul 17 19:33:22.246203 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.246166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:22.266955 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.266912    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"]
Jul 17 19:33:22.269789 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.269760    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2109fd-29738610-xtv4d"]
Jul 17 19:33:22.655501 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.655479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:33:22.799006 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.798920    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9292d38-97ce-4073-9841-00123362c521-openshift-service-ca\") pod \"f9292d38-97ce-4073-9841-00123362c521\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") "
Jul 17 19:33:22.799006 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.798949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69hpj\" (UniqueName: \"kubernetes.io/projected/f9292d38-97ce-4073-9841-00123362c521-kube-api-access-69hpj\") pod \"f9292d38-97ce-4073-9841-00123362c521\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") "
Jul 17 19:33:22.799006 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.798986    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls\") pod \"f9292d38-97ce-4073-9841-00123362c521\" (UID: \"f9292d38-97ce-4073-9841-00123362c521\") "
Jul 17 19:33:22.799416 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.799383    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9292d38-97ce-4073-9841-00123362c521-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9292d38-97ce-4073-9841-00123362c521" (UID: "f9292d38-97ce-4073-9841-00123362c521"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:33:22.801328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.801300    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9292d38-97ce-4073-9841-00123362c521-kube-api-access-69hpj" (OuterVolumeSpecName: "kube-api-access-69hpj") pod "f9292d38-97ce-4073-9841-00123362c521" (UID: "f9292d38-97ce-4073-9841-00123362c521"). InnerVolumeSpecName "kube-api-access-69hpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:33:22.801328 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.801313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9292d38-97ce-4073-9841-00123362c521" (UID: "f9292d38-97ce-4073-9841-00123362c521"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:33:22.899930 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.899900    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9292d38-97ce-4073-9841-00123362c521-openshift-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:22.899930 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.899926    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hpj\" (UniqueName: \"kubernetes.io/projected/f9292d38-97ce-4073-9841-00123362c521-kube-api-access-69hpj\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:22.899930 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:22.899936    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9292d38-97ce-4073-9841-00123362c521-maas-api-tls\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:23.239671 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.239627    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9292d38-97ce-4073-9841-00123362c521" containerID="5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d" exitCode=0
Jul 17 19:33:23.240194 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.239683    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"
Jul 17 19:33:23.240194 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.239724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg" event={"ID":"f9292d38-97ce-4073-9841-00123362c521","Type":"ContainerDied","Data":"5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d"}
Jul 17 19:33:23.240194 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.239775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg" event={"ID":"f9292d38-97ce-4073-9841-00123362c521","Type":"ContainerDied","Data":"dc80df21149f36cbebbc4eb4686d624081dd9f892a65d262dcfd983426025120"}
Jul 17 19:33:23.240194 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.239796    2563 scope.go:117] "RemoveContainer" containerID="5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d"
Jul 17 19:33:23.248235 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.248215    2563 scope.go:117] "RemoveContainer" containerID="5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d"
Jul 17 19:33:23.248498 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:33:23.248471    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d\": container with ID starting with 5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d not found: ID does not exist" containerID="5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d"
Jul 17 19:33:23.248544 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.248509    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d"} err="failed to get container status \"5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d\": rpc error: code = NotFound desc = could not find container \"5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d\": container with ID starting with 5cd9a4aaac4d81126ea2700863fa3b0233e2334557f817755c2b45900bf1880d not found: ID does not exist"
Jul 17 19:33:23.260478 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.260448    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"]
Jul 17 19:33:23.266749 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.266719    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2109fd-5f946f7fc7-mwvmg"]
Jul 17 19:33:23.694202 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.694161    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c82c1b9-9fd8-4bce-b585-c17af79f3160" path="/var/lib/kubelet/pods/0c82c1b9-9fd8-4bce-b585-c17af79f3160/volumes"
Jul 17 19:33:23.694614 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:23.694593    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9292d38-97ce-4073-9841-00123362c521" path="/var/lib/kubelet/pods/f9292d38-97ce-4073-9841-00123362c521/volumes"
Jul 17 19:33:25.310398 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:25.310362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9ctsp"]
Jul 17 19:33:26.918165 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:26.918103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:33.291487 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:33.291452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:33.352133 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:33.352094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:34.619414 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619382    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"]
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619663    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619673    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d68cbe5-64e5-4cd5-97d2-bfbc807cb003" containerName="revoke-keys"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619686    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="773daa09-4b25-466b-b70e-9b5d0d6e28e5" containerName="maas-api"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619692    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="773daa09-4b25-466b-b70e-9b5d0d6e28e5" containerName="maas-api"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619699    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619704    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619717    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619726    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9292d38-97ce-4073-9841-00123362c521" containerName="maas-api"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619731    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9292d38-97ce-4073-9841-00123362c521" containerName="maas-api"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619777    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619784    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="773daa09-4b25-466b-b70e-9b5d0d6e28e5" containerName="maas-api"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619791    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9292d38-97ce-4073-9841-00123362c521" containerName="maas-api"
Jul 17 19:33:34.619810 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.619795    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:34.623639 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.623621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"
Jul 17 19:33:34.626108 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.626080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:33:34.626268 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.626146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:33:34.629376 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.629351    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"]
Jul 17 19:33:34.690241 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.690210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27282dad-f1d0-4991-8ed6-8c10bf138874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb\" (UID: \"27282dad-f1d0-4991-8ed6-8c10bf138874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"
Jul 17 19:33:34.791500 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.791459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27282dad-f1d0-4991-8ed6-8c10bf138874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb\" (UID: \"27282dad-f1d0-4991-8ed6-8c10bf138874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"
Jul 17 19:33:34.792142 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.792102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27282dad-f1d0-4991-8ed6-8c10bf138874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb\" (UID: \"27282dad-f1d0-4991-8ed6-8c10bf138874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"
Jul 17 19:33:34.935510 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:34.935405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"
Jul 17 19:33:35.067141 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:35.067098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"]
Jul 17 19:33:35.069831 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:33:35.069804    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27282dad_f1d0_4991_8ed6_8c10bf138874.slice/crio-e06ac58b2680294ceab78944796a09e10e8875b05bd57b1d686d8c7cae6805d8 WatchSource:0}: Error finding container e06ac58b2680294ceab78944796a09e10e8875b05bd57b1d686d8c7cae6805d8: Status 404 returned error can't find the container with id e06ac58b2680294ceab78944796a09e10e8875b05bd57b1d686d8c7cae6805d8
Jul 17 19:33:35.296471 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:35.296440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" event={"ID":"27282dad-f1d0-4991-8ed6-8c10bf138874","Type":"ContainerStarted","Data":"e06ac58b2680294ceab78944796a09e10e8875b05bd57b1d686d8c7cae6805d8"}
Jul 17 19:33:36.301962 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:36.301929    2563 generic.go:356] "Generic (PLEG): container finished" podID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerID="848c5c3b5fe71f48f78bade876587e37c9f10d9ee6e6addf84b27dcdb64c3ae3" exitCode=6
Jul 17 19:33:36.302372 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:36.302025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" event={"ID":"27282dad-f1d0-4991-8ed6-8c10bf138874","Type":"ContainerDied","Data":"848c5c3b5fe71f48f78bade876587e37c9f10d9ee6e6addf84b27dcdb64c3ae3"}
Jul 17 19:33:36.302372 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:36.302193    2563 scope.go:117] "RemoveContainer" containerID="848c5c3b5fe71f48f78bade876587e37c9f10d9ee6e6addf84b27dcdb64c3ae3"
Jul 17 19:33:37.308671 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.308570    2563 generic.go:356] "Generic (PLEG): container finished" podID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerID="5c908165173a62fe4f454bda29d1cb0dd0fb79b519fd8e3aa30911f70145485d" exitCode=6
Jul 17 19:33:37.308671 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.308652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" event={"ID":"27282dad-f1d0-4991-8ed6-8c10bf138874","Type":"ContainerDied","Data":"5c908165173a62fe4f454bda29d1cb0dd0fb79b519fd8e3aa30911f70145485d"}
Jul 17 19:33:37.309178 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.308714    2563 scope.go:117] "RemoveContainer" containerID="848c5c3b5fe71f48f78bade876587e37c9f10d9ee6e6addf84b27dcdb64c3ae3"
Jul 17 19:33:37.309178 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.308965    2563 scope.go:117] "RemoveContainer" containerID="5c908165173a62fe4f454bda29d1cb0dd0fb79b519fd8e3aa30911f70145485d"
Jul 17 19:33:37.309260 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:33:37.309214    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb_odh-ai-gateway-infra(27282dad-f1d0-4991-8ed6-8c10bf138874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874"
Jul 17 19:33:37.561032 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.560943    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"]
Jul 17 19:33:37.561264 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.561234    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy" containerID="cri-o://56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89" gracePeriod=30
Jul 17 19:33:37.577321 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.577291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:37.817106 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.816994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:37.825147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.825103    2563 patch_prober.go:28] interesting pod/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 19:33:37.825298 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:37.825173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 19:33:38.315943 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:38.315917    2563 scope.go:117] "RemoveContainer" containerID="5c908165173a62fe4f454bda29d1cb0dd0fb79b519fd8e3aa30911f70145485d"
Jul 17 19:33:38.316348 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:33:38.316094    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb_odh-ai-gateway-infra(27282dad-f1d0-4991-8ed6-8c10bf138874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874"
Jul 17 19:33:42.807025 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.806997    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:33:42.860768 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.860741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df955cae-ce35-484c-9af0-a81c1e13fc5d-istiod-ca-cert\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.860924 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.860805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-envoy\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.860924 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.860834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-certs\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.860924 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.860853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-data\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.860924 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.860903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-podinfo\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.861147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.860957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w24t\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-kube-api-access-7w24t\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.861147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-socket\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.861147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-token\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.861147 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861092    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-credential-socket\") pod \"df955cae-ce35-484c-9af0-a81c1e13fc5d\" (UID: \"df955cae-ce35-484c-9af0-a81c1e13fc5d\") "
Jul 17 19:33:42.861388 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861263    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df955cae-ce35-484c-9af0-a81c1e13fc5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:33:42.861456 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861397    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df955cae-ce35-484c-9af0-a81c1e13fc5d-istiod-ca-cert\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:42.861456 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861391    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:33:42.861559 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861491    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:33:42.861559 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:33:42.861870 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.861845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:33:42.863265 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.863236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 19:33:42.863382 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.863290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-kube-api-access-7w24t" (OuterVolumeSpecName: "kube-api-access-7w24t") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "kube-api-access-7w24t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:33:42.863382 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.863318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:33:42.863545 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.863521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "df955cae-ce35-484c-9af0-a81c1e13fc5d" (UID: "df955cae-ce35-484c-9af0-a81c1e13fc5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:33:42.962638 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.962603    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:42.962638 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.962636    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-token\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:42.962638 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.962645    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-credential-socket\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:42.962638 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.962654    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-envoy\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:42.962920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.962663    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-workload-certs\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:42.962920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.962670    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-data\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:42.962920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.962677    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df955cae-ce35-484c-9af0-a81c1e13fc5d-istio-podinfo\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:42.962920 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:42.962685    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w24t\" (UniqueName: \"kubernetes.io/projected/df955cae-ce35-484c-9af0-a81c1e13fc5d-kube-api-access-7w24t\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:43.341026 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.340926    2563 generic.go:356] "Generic (PLEG): container finished" podID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerID="56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89" exitCode=0
Jul 17 19:33:43.341026 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.340983    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"
Jul 17 19:33:43.341026 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.340993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" event={"ID":"df955cae-ce35-484c-9af0-a81c1e13fc5d","Type":"ContainerDied","Data":"56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89"}
Jul 17 19:33:43.341371 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.341036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6" event={"ID":"df955cae-ce35-484c-9af0-a81c1e13fc5d","Type":"ContainerDied","Data":"fc00c33c42dd9cc762494f40fc61ef9eef3c48755d1d86d2068b54f0faa522bc"}
Jul 17 19:33:43.341371 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.341056    2563 scope.go:117] "RemoveContainer" containerID="56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89"
Jul 17 19:33:43.350658 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.350637    2563 scope.go:117] "RemoveContainer" containerID="f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92"
Jul 17 19:33:43.358608 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.358588    2563 scope.go:117] "RemoveContainer" containerID="56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89"
Jul 17 19:33:43.358881 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:33:43.358859    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89\": container with ID starting with 56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89 not found: ID does not exist" containerID="56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89"
Jul 17 19:33:43.358949 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.358889    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89"} err="failed to get container status \"56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89\": rpc error: code = NotFound desc = could not find container \"56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89\": container with ID starting with 56823c67336b84ec915e6e6bea3a76c28c3b8c465b638be8cd737486cea56c89 not found: ID does not exist"
Jul 17 19:33:43.358949 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.358908    2563 scope.go:117] "RemoveContainer" containerID="f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92"
Jul 17 19:33:43.359192 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:33:43.359173    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92\": container with ID starting with f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92 not found: ID does not exist" containerID="f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92"
Jul 17 19:33:43.359231 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.359201    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92"} err="failed to get container status \"f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92\": rpc error: code = NotFound desc = could not find container \"f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92\": container with ID starting with f0c4c16eccd13e6d1b4d1c1f0998ae8f431538591108aa640e2d6be055420c92 not found: ID does not exist"
Jul 17 19:33:43.364018 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.363995    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"]
Jul 17 19:33:43.368528 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.368508    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-2109fd-openshift-default-64b696cf75-56gk6"]
Jul 17 19:33:43.694312 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:43.694269    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" path="/var/lib/kubelet/pods/df955cae-ce35-484c-9af0-a81c1e13fc5d/volumes"
Jul 17 19:33:44.125939 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:44.125918    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-bjjdw_9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d/copy-manifests/0.log"
Jul 17 19:33:44.132981 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:44.132960    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-bjjdw_9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d/manager/0.log"
Jul 17 19:33:45.067078 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:45.067049    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738610-b9pqp_790a7429-f6bc-4e60-9df7-40d09a00e647/cleanup/0.log"
Jul 17 19:33:45.610214 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:45.610186    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-4aee7c-kdh4s_8dfd88b1-4aff-4e5f-a4f2-f9398f363270/revoke-keys/0.log"
Jul 17 19:33:45.822747 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:45.822720    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb_27282dad-f1d0-4991-8ed6-8c10bf138874/revoke-keys/1.log"
Jul 17 19:33:48.287240 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:48.287212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9ctsp_87518cbb-e489-4491-852f-819eca0aa98b/limitador/0.log"
Jul 17 19:33:48.870939 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:48.870906    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dbvjc_b227c4dd-4337-402e-a5e6-5fa1d383a6da/discovery/0.log"
Jul 17 19:33:50.338966 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:50.338931    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-9pnpb_baf5cd24-999a-4bce-8f38-1424810ed746/main/0.log"
Jul 17 19:33:50.689527 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:50.689498    2563 scope.go:117] "RemoveContainer" containerID="5c908165173a62fe4f454bda29d1cb0dd0fb79b519fd8e3aa30911f70145485d"
Jul 17 19:33:51.378164 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:51.378110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" event={"ID":"27282dad-f1d0-4991-8ed6-8c10bf138874","Type":"ContainerStarted","Data":"a04d87f2113179120757f78cdfa1af22fe22a4d4585df28c366f296100de3b29"}
Jul 17 19:33:51.394521 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:51.394470    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" podStartSLOduration=17.394454869 podStartE2EDuration="17.394454869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:51.393330848 +0000 UTC m=+3640.304157609" watchObservedRunningTime="2026-07-17 19:33:51.394454869 +0000 UTC m=+3640.305281629"
Jul 17 19:33:52.384417 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:52.384382    2563 generic.go:356] "Generic (PLEG): container finished" podID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerID="a04d87f2113179120757f78cdfa1af22fe22a4d4585df28c366f296100de3b29" exitCode=6
Jul 17 19:33:52.384863 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:52.384433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" event={"ID":"27282dad-f1d0-4991-8ed6-8c10bf138874","Type":"ContainerDied","Data":"a04d87f2113179120757f78cdfa1af22fe22a4d4585df28c366f296100de3b29"}
Jul 17 19:33:52.384863 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:52.384492    2563 scope.go:117] "RemoveContainer" containerID="5c908165173a62fe4f454bda29d1cb0dd0fb79b519fd8e3aa30911f70145485d"
Jul 17 19:33:52.384863 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:52.384744    2563 scope.go:117] "RemoveContainer" containerID="a04d87f2113179120757f78cdfa1af22fe22a4d4585df28c366f296100de3b29"
Jul 17 19:33:52.384986 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:33:52.384970    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb_odh-ai-gateway-infra(27282dad-f1d0-4991-8ed6-8c10bf138874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874"
Jul 17 19:33:52.406145 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:52.406103    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"]
Jul 17 19:33:53.506871 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.506842    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"]
Jul 17 19:33:53.507216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507158    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy"
Jul 17 19:33:53.507216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507171    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy"
Jul 17 19:33:53.507216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507185    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy"
Jul 17 19:33:53.507216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507190    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy"
Jul 17 19:33:53.507216 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507216    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:53.507406 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507222    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:53.507406 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507269    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy"
Jul 17 19:33:53.507406 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507278    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="03606466-a539-4213-b88c-619fa94ca713" containerName="revoke-keys"
Jul 17 19:33:53.507406 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.507288    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="df955cae-ce35-484c-9af0-a81c1e13fc5d" containerName="istio-proxy"
Jul 17 19:33:53.510626 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.510609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"
Jul 17 19:33:53.518180 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.518141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"]
Jul 17 19:33:53.526392 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.526308    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"
Jul 17 19:33:53.551067 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.551033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1be64f-271b-44eb-8cfb-eb41815f55bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7\" (UID: \"6c1be64f-271b-44eb-8cfb-eb41815f55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"
Jul 17 19:33:53.651994 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.651903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27282dad-f1d0-4991-8ed6-8c10bf138874-maas-api-service-ca\") pod \"27282dad-f1d0-4991-8ed6-8c10bf138874\" (UID: \"27282dad-f1d0-4991-8ed6-8c10bf138874\") "
Jul 17 19:33:53.652170 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.652092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1be64f-271b-44eb-8cfb-eb41815f55bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7\" (UID: \"6c1be64f-271b-44eb-8cfb-eb41815f55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"
Jul 17 19:33:53.652332 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.652313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27282dad-f1d0-4991-8ed6-8c10bf138874-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27282dad-f1d0-4991-8ed6-8c10bf138874" (UID: "27282dad-f1d0-4991-8ed6-8c10bf138874"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:33:53.652769 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.652747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1be64f-271b-44eb-8cfb-eb41815f55bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7\" (UID: \"6c1be64f-271b-44eb-8cfb-eb41815f55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"
Jul 17 19:33:53.752975 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.752935    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27282dad-f1d0-4991-8ed6-8c10bf138874-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:33:53.832794 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.832760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"
Jul 17 19:33:53.960135 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:53.960090    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"]
Jul 17 19:33:53.963025 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:33:53.962995    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1be64f_271b_44eb_8cfb_eb41815f55bc.slice/crio-64f146903ce2f152575db2a064761c472aa15bff141858f31d7090a5b58199ac WatchSource:0}: Error finding container 64f146903ce2f152575db2a064761c472aa15bff141858f31d7090a5b58199ac: Status 404 returned error can't find the container with id 64f146903ce2f152575db2a064761c472aa15bff141858f31d7090a5b58199ac
Jul 17 19:33:54.396168 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:54.396133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" event={"ID":"6c1be64f-271b-44eb-8cfb-eb41815f55bc","Type":"ContainerStarted","Data":"4ee459a4fcdc6f9a137bcad2f2eef4ae75090f6721e082e8c4362f372ed1d1f9"}
Jul 17 19:33:54.396361 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:54.396195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" event={"ID":"6c1be64f-271b-44eb-8cfb-eb41815f55bc","Type":"ContainerStarted","Data":"64f146903ce2f152575db2a064761c472aa15bff141858f31d7090a5b58199ac"}
Jul 17 19:33:54.397404 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:54.397382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb" event={"ID":"27282dad-f1d0-4991-8ed6-8c10bf138874","Type":"ContainerDied","Data":"e06ac58b2680294ceab78944796a09e10e8875b05bd57b1d686d8c7cae6805d8"}
Jul 17 19:33:54.397520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:54.397406    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"
Jul 17 19:33:54.397520 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:54.397431    2563 scope.go:117] "RemoveContainer" containerID="a04d87f2113179120757f78cdfa1af22fe22a4d4585df28c366f296100de3b29"
Jul 17 19:33:54.412628 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:54.412586    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" podStartSLOduration=1.412571422 podStartE2EDuration="1.412571422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:54.411308555 +0000 UTC m=+3643.322135336" watchObservedRunningTime="2026-07-17 19:33:54.412571422 +0000 UTC m=+3643.323398183"
Jul 17 19:33:54.424433 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:54.424409    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"]
Jul 17 19:33:54.427912 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:54.427887    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-kt8nb"]
Jul 17 19:33:55.403553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:55.403517    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerID="4ee459a4fcdc6f9a137bcad2f2eef4ae75090f6721e082e8c4362f372ed1d1f9" exitCode=6
Jul 17 19:33:55.403553 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:55.403557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" event={"ID":"6c1be64f-271b-44eb-8cfb-eb41815f55bc","Type":"ContainerDied","Data":"4ee459a4fcdc6f9a137bcad2f2eef4ae75090f6721e082e8c4362f372ed1d1f9"}
Jul 17 19:33:55.403997 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:55.403787    2563 scope.go:117] "RemoveContainer" containerID="4ee459a4fcdc6f9a137bcad2f2eef4ae75090f6721e082e8c4362f372ed1d1f9"
Jul 17 19:33:55.693613 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:55.693588    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" path="/var/lib/kubelet/pods/27282dad-f1d0-4991-8ed6-8c10bf138874/volumes"
Jul 17 19:33:56.050676 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:56.050593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:56.109665 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:56.109634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:33:56.415012 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:56.414971    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerID="fea3b504ce678ea7c87f210a493ac3b27c892e46f0502e31b5d124461eaf5bc8" exitCode=6
Jul 17 19:33:56.415523 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:56.415056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" event={"ID":"6c1be64f-271b-44eb-8cfb-eb41815f55bc","Type":"ContainerDied","Data":"fea3b504ce678ea7c87f210a493ac3b27c892e46f0502e31b5d124461eaf5bc8"}
Jul 17 19:33:56.415523 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:56.415200    2563 scope.go:117] "RemoveContainer" containerID="4ee459a4fcdc6f9a137bcad2f2eef4ae75090f6721e082e8c4362f372ed1d1f9"
Jul 17 19:33:56.415523 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:56.415356    2563 scope.go:117] "RemoveContainer" containerID="fea3b504ce678ea7c87f210a493ac3b27c892e46f0502e31b5d124461eaf5bc8"
Jul 17 19:33:56.415686 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:33:56.415610    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7_odh-ai-gateway-infra(6c1be64f-271b-44eb-8cfb-eb41815f55bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc"
Jul 17 19:33:57.421711 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:57.421688    2563 scope.go:117] "RemoveContainer" containerID="fea3b504ce678ea7c87f210a493ac3b27c892e46f0502e31b5d124461eaf5bc8"
Jul 17 19:33:57.422147 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:33:57.421881    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7_odh-ai-gateway-infra(6c1be64f-271b-44eb-8cfb-eb41815f55bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc"
Jul 17 19:33:58.866596 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:58.866566    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t88vb_f0baa97c-7dca-43b5-b52d-680c32593d70/global-pull-secret-syncer/0.log"
Jul 17 19:33:58.918286 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:58.918255    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log"
Jul 17 19:33:59.054073 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:33:59.054045    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-130.ec2.internal_1f0ff61e2ec63402244f77cf1bd9480b/haproxy/0.log"
Jul 17 19:34:04.579554 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:04.579525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9ctsp_87518cbb-e489-4491-852f-819eca0aa98b/limitador/0.log"
Jul 17 19:34:06.771617 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:06.771579    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frrn9_282366ea-d550-407f-a397-81dafba91392/node-exporter/0.log"
Jul 17 19:34:06.796447 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:06.796371    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frrn9_282366ea-d550-407f-a397-81dafba91392/kube-rbac-proxy/0.log"
Jul 17 19:34:06.822564 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:06.822538    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-frrn9_282366ea-d550-407f-a397-81dafba91392/init-textfile/0.log"
Jul 17 19:34:07.075217 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:07.075138    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_016bbbfb-9726-42d2-9ad9-6485660e5edd/prometheus/0.log"
Jul 17 19:34:07.095924 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:07.095897    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_016bbbfb-9726-42d2-9ad9-6485660e5edd/config-reloader/0.log"
Jul 17 19:34:07.116511 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:07.116473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_016bbbfb-9726-42d2-9ad9-6485660e5edd/thanos-sidecar/0.log"
Jul 17 19:34:07.136281 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:07.136256    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_016bbbfb-9726-42d2-9ad9-6485660e5edd/kube-rbac-proxy-web/0.log"
Jul 17 19:34:07.157704 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:07.157679    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_016bbbfb-9726-42d2-9ad9-6485660e5edd/kube-rbac-proxy/0.log"
Jul 17 19:34:07.178555 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:07.178529    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_016bbbfb-9726-42d2-9ad9-6485660e5edd/kube-rbac-proxy-thanos/0.log"
Jul 17 19:34:07.201096 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:07.201069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_016bbbfb-9726-42d2-9ad9-6485660e5edd/init-config-reloader/0.log"
Jul 17 19:34:08.587295 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587259    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"]
Jul 17 19:34:08.587666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587557    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.587666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587568    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.587666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587581    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.587666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587587    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.587666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.587666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587599    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.587666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587658    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.587666 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587668    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.587919 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.587674    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27282dad-f1d0-4991-8ed6-8c10bf138874" containerName="revoke-keys"
Jul 17 19:34:08.591826 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.591802    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.594289 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.594263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dtqk6\"/\"default-dockercfg-n4gpm\""
Jul 17 19:34:08.594289 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.594275    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtqk6\"/\"openshift-service-ca.crt\""
Jul 17 19:34:08.595394 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.595372    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtqk6\"/\"kube-root-ca.crt\""
Jul 17 19:34:08.598458 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.598431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"]
Jul 17 19:34:08.681279 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.681241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-lib-modules\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.681279 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.681278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-proc\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.681491 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.681298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7n7\" (UniqueName: \"kubernetes.io/projected/dfbe7101-b220-47c5-aaea-850f04664e58-kube-api-access-8k7n7\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.681491 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.681408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-sys\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.681491 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.681474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-podres\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782430 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-podres\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782577 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-lib-modules\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782577 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-proc\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782577 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7n7\" (UniqueName: \"kubernetes.io/projected/dfbe7101-b220-47c5-aaea-850f04664e58-kube-api-access-8k7n7\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782577 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-sys\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782730 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-lib-modules\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782730 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-podres\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782730 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-proc\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.782730 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.782620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfbe7101-b220-47c5-aaea-850f04664e58-sys\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.792037 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.792003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7n7\" (UniqueName: \"kubernetes.io/projected/dfbe7101-b220-47c5-aaea-850f04664e58-kube-api-access-8k7n7\") pod \"perf-node-gather-daemonset-jwjx6\" (UID: \"dfbe7101-b220-47c5-aaea-850f04664e58\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:08.903443 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:08.903409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:09.040275 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:09.040238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"]
Jul 17 19:34:09.043514 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:34:09.043488    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddfbe7101_b220_47c5_aaea_850f04664e58.slice/crio-22e6c8936fd2bfb13d9e3c0b10bc50644993241d8b69cb4a429ffc4ecd8ffae3 WatchSource:0}: Error finding container 22e6c8936fd2bfb13d9e3c0b10bc50644993241d8b69cb4a429ffc4ecd8ffae3: Status 404 returned error can't find the container with id 22e6c8936fd2bfb13d9e3c0b10bc50644993241d8b69cb4a429ffc4ecd8ffae3
Jul 17 19:34:09.477898 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:09.477811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6" event={"ID":"dfbe7101-b220-47c5-aaea-850f04664e58","Type":"ContainerStarted","Data":"df7a01dbfb176426c53319c09bb5c89c5ee637545f95e756cbf9cfecb9f9f2d4"}
Jul 17 19:34:09.477898 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:09.477865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6" event={"ID":"dfbe7101-b220-47c5-aaea-850f04664e58","Type":"ContainerStarted","Data":"22e6c8936fd2bfb13d9e3c0b10bc50644993241d8b69cb4a429ffc4ecd8ffae3"}
Jul 17 19:34:09.478093 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:09.477924    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:09.496102 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:09.496044    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6" podStartSLOduration=1.496026453 podStartE2EDuration="1.496026453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:34:09.494206159 +0000 UTC m=+3658.405032923" watchObservedRunningTime="2026-07-17 19:34:09.496026453 +0000 UTC m=+3658.406853213"
Jul 17 19:34:11.474467 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:11.474436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7dccw_f00f4c0f-381e-4dac-8330-c769a42e1b88/dns/0.log"
Jul 17 19:34:11.494759 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:11.494735    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7dccw_f00f4c0f-381e-4dac-8330-c769a42e1b88/kube-rbac-proxy/0.log"
Jul 17 19:34:11.611655 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:11.611613    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dlf6l_7d3069f6-5dc2-40cd-9459-917d4358b97c/dns-node-resolver/0.log"
Jul 17 19:34:12.229000 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:12.228966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b7d76964d-pht9r_199cb7af-c161-4c54-997f-8d8f8476c9a5/registry/0.log"
Jul 17 19:34:12.254367 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:12.254336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lsgfn_37c764c7-9cd9-40cd-8363-87505d4d1df3/node-ca/0.log"
Jul 17 19:34:12.688958 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:12.688929    2563 scope.go:117] "RemoveContainer" containerID="fea3b504ce678ea7c87f210a493ac3b27c892e46f0502e31b5d124461eaf5bc8"
Jul 17 19:34:13.379900 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:13.379865    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dbvjc_b227c4dd-4337-402e-a5e6-5fa1d383a6da/discovery/0.log"
Jul 17 19:34:13.499532 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:13.499500    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerID="a5d2d68973dd38a34ddb4cc06e0b3dd483e6878cc3b0a9187fc650e37678bf24" exitCode=6
Jul 17 19:34:13.499716 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:13.499580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" event={"ID":"6c1be64f-271b-44eb-8cfb-eb41815f55bc","Type":"ContainerDied","Data":"a5d2d68973dd38a34ddb4cc06e0b3dd483e6878cc3b0a9187fc650e37678bf24"}
Jul 17 19:34:13.499716 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:13.499643    2563 scope.go:117] "RemoveContainer" containerID="fea3b504ce678ea7c87f210a493ac3b27c892e46f0502e31b5d124461eaf5bc8"
Jul 17 19:34:13.499964 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:13.499948    2563 scope.go:117] "RemoveContainer" containerID="a5d2d68973dd38a34ddb4cc06e0b3dd483e6878cc3b0a9187fc650e37678bf24"
Jul 17 19:34:13.500174 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:34:13.500152    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7_odh-ai-gateway-infra(6c1be64f-271b-44eb-8cfb-eb41815f55bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc"
Jul 17 19:34:14.307633 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:14.307607    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tqqs_df37e276-43e7-41eb-88e6-4f78efcbcd7d/serve-healthcheck-canary/0.log"
Jul 17 19:34:14.532738 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:14.532703    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"]
Jul 17 19:34:14.664860 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:14.664835    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"
Jul 17 19:34:14.838634 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:14.838519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1be64f-271b-44eb-8cfb-eb41815f55bc-maas-api-service-ca\") pod \"6c1be64f-271b-44eb-8cfb-eb41815f55bc\" (UID: \"6c1be64f-271b-44eb-8cfb-eb41815f55bc\") "
Jul 17 19:34:14.838958 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:14.838937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1be64f-271b-44eb-8cfb-eb41815f55bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c1be64f-271b-44eb-8cfb-eb41815f55bc" (UID: "6c1be64f-271b-44eb-8cfb-eb41815f55bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:34:14.939423 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:14.939391    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1be64f-271b-44eb-8cfb-eb41815f55bc-maas-api-service-ca\") on node \"ip-10-0-138-130.ec2.internal\" DevicePath \"\""
Jul 17 19:34:15.008277 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.008252    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sr4bl_6c50c47d-872f-4c0f-8784-793d22823ecd/kube-rbac-proxy/0.log"
Jul 17 19:34:15.027869 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.027844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sr4bl_6c50c47d-872f-4c0f-8784-793d22823ecd/exporter/0.log"
Jul 17 19:34:15.049091 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.049068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sr4bl_6c50c47d-872f-4c0f-8784-793d22823ecd/extractor/0.log"
Jul 17 19:34:15.492716 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.492685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-jwjx6"
Jul 17 19:34:15.511381 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.511353    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"
Jul 17 19:34:15.511550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.511352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7" event={"ID":"6c1be64f-271b-44eb-8cfb-eb41815f55bc","Type":"ContainerDied","Data":"64f146903ce2f152575db2a064761c472aa15bff141858f31d7090a5b58199ac"}
Jul 17 19:34:15.511550 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.511483    2563 scope.go:117] "RemoveContainer" containerID="a5d2d68973dd38a34ddb4cc06e0b3dd483e6878cc3b0a9187fc650e37678bf24"
Jul 17 19:34:15.530609 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.530580    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"]
Jul 17 19:34:15.536388 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.536363    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-k6vp7"]
Jul 17 19:34:15.693763 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:15.693730    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" path="/var/lib/kubelet/pods/6c1be64f-271b-44eb-8cfb-eb41815f55bc/volumes"
Jul 17 19:34:16.470796 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.470752    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8"]
Jul 17 19:34:16.471107 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.471094    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:16.471188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.471110    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:16.471188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.471133    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:16.471188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.471139    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:16.471188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.471150    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:16.471188 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.471156    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:16.471351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.471208    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:16.471351 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.471216    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:16.475659 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.475639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8"
Jul 17 19:34:16.478460 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.478437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:34:16.478590 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.478461    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:34:16.482095 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.482067    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8"]
Jul 17 19:34:16.654434 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.654403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8\" (UID: \"cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8"
Jul 17 19:34:16.755162 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.755044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8\" (UID: \"cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8"
Jul 17 19:34:16.755762 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.755732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8\" (UID: \"cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8"
Jul 17 19:34:16.787163 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.787107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8"
Jul 17 19:34:16.915904 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:16.915866    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8"]
Jul 17 19:34:16.919413 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:34:16.919376    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8f518c_ed4f_415e_ba42_5eb2eb9da6f5.slice/crio-ff350faaeda4bfd7e7b67d10cce47ad6ebfa82ed6066a5c1619ce35aae5305c1 WatchSource:0}: Error finding container ff350faaeda4bfd7e7b67d10cce47ad6ebfa82ed6066a5c1619ce35aae5305c1: Status 404 returned error can't find the container with id ff350faaeda4bfd7e7b67d10cce47ad6ebfa82ed6066a5c1619ce35aae5305c1
Jul 17 19:34:17.522392 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:17.522358    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5" containerID="08f8e96939a4d2e661a65fcd4dc3912d966f42e1dff2cd3926f156aa94946008" exitCode=6
Jul 17 19:34:17.522582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:17.522473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8" event={"ID":"cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5","Type":"ContainerDied","Data":"08f8e96939a4d2e661a65fcd4dc3912d966f42e1dff2cd3926f156aa94946008"}
Jul 17 19:34:17.522582 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:17.522514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8" event={"ID":"cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5","Type":"ContainerStarted","Data":"ff350faaeda4bfd7e7b67d10cce47ad6ebfa82ed6066a5c1619ce35aae5305c1"}
Jul 17 19:34:17.522691 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:17.522624    2563 scope.go:117] "RemoveContainer" containerID="08f8e96939a4d2e661a65fcd4dc3912d966f42e1dff2cd3926f156aa94946008"
Jul 17 19:34:17.818375 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:17.818345    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-bjjdw_9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d/manager/0.log"
Jul 17 19:34:17.841935 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:17.841905    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-bjjdw_9fc7b91d-6fdd-44fc-8ac0-d1594aec5e5d/copy-manifests/0.log"
Jul 17 19:34:18.528790 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:18.528699    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5" containerID="45ed2726996279c7200188a35c2c1c840aa7dbb43a7e9f7022aa8e9854378af5" exitCode=6
Jul 17 19:34:18.528790 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:18.528753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8" event={"ID":"cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5","Type":"ContainerDied","Data":"45ed2726996279c7200188a35c2c1c840aa7dbb43a7e9f7022aa8e9854378af5"}
Jul 17 19:34:18.529011 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:18.528805    2563 scope.go:117] "RemoveContainer" containerID="08f8e96939a4d2e661a65fcd4dc3912d966f42e1dff2cd3926f156aa94946008"
Jul 17 19:34:18.529011 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:18.528992    2563 scope.go:117] "RemoveContainer" containerID="45ed2726996279c7200188a35c2c1c840aa7dbb43a7e9f7022aa8e9854378af5"
Jul 17 19:34:18.529256 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:34:18.529233    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8_odh-ai-gateway-infra(cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8" podUID="cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5"
Jul 17 19:34:19.195865 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:19.195835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dbvjc_b227c4dd-4337-402e-a5e6-5fa1d383a6da/discovery/0.log"
Jul 17 19:34:19.373525 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:19.373498    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-cbc4544f6-dl9ct_c2b38ddf-00a1-4c6a-a20b-5f3b480c7055/manager/0.log"
Jul 17 19:34:19.534584 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:19.534497    2563 scope.go:117] "RemoveContainer" containerID="45ed2726996279c7200188a35c2c1c840aa7dbb43a7e9f7022aa8e9854378af5"
Jul 17 19:34:19.534754 ip-10-0-138-130 kubenswrapper[2563]: E0717 19:34:19.534720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8_odh-ai-gateway-infra(cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-hrql8" podUID="cc8f518c-ed4f-415e-ba42-5eb2eb9da6f5"
Jul 17 19:34:20.676044 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:20.676006    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dbvjc_b227c4dd-4337-402e-a5e6-5fa1d383a6da/discovery/0.log"
Jul 17 19:34:23.201852 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:23.201814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ltlgb_73a4619f-1e89-431b-a954-b857d9810083/konnectivity-agent/0.log" limit=1024
Jul 17 19:34:24.661862 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:24.661819    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl"]
Jul 17 19:34:24.662280 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:24.662214    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1be64f-271b-44eb-8cfb-eb41815f55bc" containerName="revoke-keys"
Jul 17 19:34:24.670942 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:24.670919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl"
Jul 17 19:34:24.672106 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:24.672074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl"]
Jul 17 19:34:24.826260 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:24.826217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f34cb9-6fca-4307-aedb-fd73c8de8ba2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl\" (UID: \"e7f34cb9-6fca-4307-aedb-fd73c8de8ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl"
Jul 17 19:34:24.927567 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:24.927456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f34cb9-6fca-4307-aedb-fd73c8de8ba2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl\" (UID: \"e7f34cb9-6fca-4307-aedb-fd73c8de8ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl"
Jul 17 19:34:24.928142 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:24.928090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f34cb9-6fca-4307-aedb-fd73c8de8ba2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl\" (UID: \"e7f34cb9-6fca-4307-aedb-fd73c8de8ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl"
Jul 17 19:34:24.981618 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:24.981578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl"
Jul 17 19:34:25.114185 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:25.114149    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl"]
Jul 17 19:34:25.117637 ip-10-0-138-130 kubenswrapper[2563]: W0717 19:34:25.117607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f34cb9_6fca_4307_aedb_fd73c8de8ba2.slice/crio-0f5c95249c2ef7aefed4fbb1ff2cbaacd670247ae0ea676b7d906b43fd5150e7 WatchSource:0}: Error finding container 0f5c95249c2ef7aefed4fbb1ff2cbaacd670247ae0ea676b7d906b43fd5150e7: Status 404 returned error can't find the container with id 0f5c95249c2ef7aefed4fbb1ff2cbaacd670247ae0ea676b7d906b43fd5150e7
Jul 17 19:34:25.561106 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:25.561002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl" event={"ID":"e7f34cb9-6fca-4307-aedb-fd73c8de8ba2","Type":"ContainerStarted","Data":"3a27e278d1e1d76d050c261af49be3433fdc3266547b18616b2bc5213b40c11e"}
Jul 17 19:34:25.561106 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:25.561068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl" event={"ID":"e7f34cb9-6fca-4307-aedb-fd73c8de8ba2","Type":"ContainerStarted","Data":"0f5c95249c2ef7aefed4fbb1ff2cbaacd670247ae0ea676b7d906b43fd5150e7"}
Jul 17 19:34:25.575717 ip-10-0-138-130 kubenswrapper[2563]: I0717 19:34:25.575661    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-md5pl" podStartSLOduration=1.575644576 podStartE2EDuration="1.575644576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:34:25.574764738 +0000 UTC m=+3674.485591500" watchObservedRunningTime="2026-07-17 19:34:25.575644576 +0000 UTC m=+3674.486471337"