Jul 18 00:40:29.566345 ip-10-0-143-142 systemd[1]: Starting Kubernetes Kubelet...
Jul 18 00:40:29.941713 ip-10-0-143-142 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 00:40:29.941713 ip-10-0-143-142 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 18 00:40:29.941713 ip-10-0-143-142 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 00:40:29.941713 ip-10-0-143-142 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 18 00:40:29.941713 ip-10-0-143-142 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 00:40:29.941713 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.940594    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946202    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946215    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946219    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946222    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946225    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946229    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946232    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946234    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946237    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946240    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946243    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946245    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946248    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946251    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946254    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946256    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946266    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946270    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946273    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:29.948992 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946275    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946278    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946281    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946283    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946286    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946289    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946291    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946294    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946296    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946301    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946305    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946308    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946311    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946314    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946317    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946320    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946323    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946326    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946329    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:29.950478 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946331    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946334    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946337    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946340    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946342    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946345    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946348    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946351    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946354    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946358    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946362    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946366    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946371    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946374    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946378    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946381    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946384    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946388    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946392    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946395    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:29.954537 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946399    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946402    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946405    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946408    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946412    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946416    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946420    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946423    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946426    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946429    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946432    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946436    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946439    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946443    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946447    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946451    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946455    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946459    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946463    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:29.955162 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946467    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946471    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946477    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946481    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946485    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946490    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946495    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946499    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946502    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946508    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946512    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946520    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946526    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946531    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946536    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946542    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946548    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946553    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946556    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:29.955648 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946561    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946565    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.946569    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947472    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947482    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947486    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947491    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947496    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947500    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947504    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947509    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947514    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947518    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947522    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947526    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947530    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947536    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947541    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947545    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947549    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:29.956186 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947553    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947557    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947561    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947564    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947567    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947570    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947574    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947579    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947583    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947586    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947589    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947593    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947596    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947599    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947603    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947606    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947609    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947612    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947616    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:29.956718 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947619    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947623    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947626    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947630    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947633    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947636    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947639    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947643    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947647    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947650    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947655    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947659    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947662    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947665    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947669    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947672    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947676    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947679    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947684    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:29.957304 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947690    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947696    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947701    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947706    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947711    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947715    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947719    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947724    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947728    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947732    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947736    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947740    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947744    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947748    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947751    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947754    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947758    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947762    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947765    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947769    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:29.957804 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947772    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947775    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947780    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947783    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947789    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947793    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947797    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947801    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947804    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947807    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947810    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947813    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947816    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947820    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947823    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947826    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947830    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947833    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947836    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947840    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:30.060124 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947843    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:30.012229 ip-10-0-143-142 systemd[1]: Started Kubernetes Kubelet.
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947846    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947850    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.947853    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948380    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948387    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948391    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948395    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948399    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948403    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948407    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948415    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948419    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948422    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948426    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948429    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948434    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948437    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948440    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:30.061072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948446    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948452    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948457    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948460    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948464    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948467    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948470    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948474    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948477    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948480    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948483    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948486    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948489    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948492    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948496    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948500    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948503    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948506    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948509    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:30.062469 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948512    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948517    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948521    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948526    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948531    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948536    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948540    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948544    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948548    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948552    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948556    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948561    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948565    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948571    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948575    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948579    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948582    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948585    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948588    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:30.063253 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948591    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948594    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948597    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948600    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948603    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948607    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948610    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948613    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948616    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948619    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948622    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948625    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948629    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948632    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948635    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948638    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948643    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948646    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948650    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948654    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:30.064072 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948657    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948660    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948664    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948667    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948670    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948674    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948678    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948682    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948685    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948688    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948691    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948694    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948697    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948701    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948703    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948707    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948710    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948713    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948716    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948719    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:30.064844 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948722    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948725    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948729    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948733    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948737    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.948741    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948841    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948850    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948859    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948864    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948871    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948875    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948880    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948885    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948889    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948893    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948898    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948903    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948909    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948913    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948916    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948920    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948923    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 18 00:40:30.065876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948927    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948936    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948941    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948946    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948950    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948956    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948962    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948967    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948972    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948978    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948983    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948987    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948991    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.948996    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949001    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949006    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949011    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949015    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949020    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949026    2563 flags.go:64] FLAG: --enable-server="true"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949030    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949038    2563 flags.go:64] FLAG: --event-burst="100"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949043    2563 flags.go:64] FLAG: --event-qps="50"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949048    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 18 00:40:30.066899 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949069    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949075    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949081    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949087    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949093    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949099    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949103    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949108    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949113    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949118    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949123    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949127    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949132    2563 flags.go:64] FLAG: --feature-gates=""
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949138    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949143    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949148    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949153    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949159    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949164    2563 flags.go:64] FLAG: --help="false"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949169    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949173    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949178    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949183    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949189    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 18 00:40:30.067659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949194    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949199    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949204    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949209    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949213    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949218    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949223    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949228    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949233    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949238    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949243    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949250    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949255    2563 flags.go:64] FLAG: --lock-file=""
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949260    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949266    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949271    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949280    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949284    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949289    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949294    2563 flags.go:64] FLAG: --logging-format="text"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949299    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949305    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949309    2563 flags.go:64] FLAG: --manifest-url=""
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949314    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949321    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 18 00:40:30.068414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949326    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949332    2563 flags.go:64] FLAG: --max-pods="110"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949337    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949342    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949346    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949351    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949356    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949360    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949365    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949377    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949382    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949387    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949392    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949397    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949405    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949410    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949416    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949421    2563 flags.go:64] FLAG: --port="10250"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949428    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949433    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02f21b38f83190b8b"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949438    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949444    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949449    2563 flags.go:64] FLAG: --register-node="true"
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949453    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 18 00:40:30.069190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949460    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949464    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949469    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949474    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949480    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949485    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949489    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949494    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949499    2563 flags.go:64] FLAG: --runonce="false"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949503    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949509    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949513    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949518    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949523    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949528    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949533    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949538    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949543    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949547    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949552    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949556    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949561    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949566    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949570    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949581    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 18 00:40:30.069810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949586    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949593    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949599    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949604    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949609    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949614    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949620    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949625    2563 flags.go:64] FLAG: --v="2"
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949631    2563 flags.go:64] FLAG: --version="false"
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949638    2563 flags.go:64] FLAG: --vmodule=""
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949643    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.949648    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949860    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949868    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949873    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949877    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949882    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949886    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949890    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949895    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949899    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949904    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:30.070502 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949908    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949913    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949921    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949927    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949931    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949935    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949939    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949942    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949945    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949950    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949954    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949958    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949961    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949965    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949968    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949972    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949976    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949980    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949983    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949987    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:30.071405 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949991    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949995    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.949998    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950001    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950005    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950008    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950011    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950014    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950018    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950021    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950024    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950028    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950032    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950036    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950039    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950042    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950045    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950049    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950069    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:30.071980 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950072    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950076    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950081    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950084    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950087    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950091    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950094    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950097    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950101    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950104    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950108    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950111    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950115    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950118    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950121    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950124    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950127    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950130    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950134    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950137    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:30.221443 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950140    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950143    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950146    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950149    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950152    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950157    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950161    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950164    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950167    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950170    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950174    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950178    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950182    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950185    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950191    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950194    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950199    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950202    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950206    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:30.245033 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950209    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950213    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950217    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950221    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950225    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950230    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950233    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950237    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950241    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950244    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.950248    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.950922    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.958249    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.958266    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 18 00:40:30.245669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.958288    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.958298    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958353    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958358    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958362    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958365    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958368    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958371    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958373    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958376    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958379    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958382    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958384    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958387    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958390    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958393    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958395    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958398    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958401    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:30.246142 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958404    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958407    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958410    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958413    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958415    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958418    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958420    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958423    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958426    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958429    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958431    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958434    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958438    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958440    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958443    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958445    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958449    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958454    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958457    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958460    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:30.246664 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958464    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958467    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958470    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958473    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958476    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958480    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958484    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958487    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958490    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958493    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958496    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958499    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958501    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958504    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958507    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958509    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958512    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958514    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958517    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958519    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:30.247257 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958522    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958525    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958527    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958530    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958532    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958535    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958538    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958541    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958544    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958546    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958549    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958551    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958554    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958556    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958558    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958561    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958563    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958566    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958569    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958571    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:30.247794 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958574    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958576    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958579    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958582    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958584    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958587    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958589    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958592    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958595    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958597    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958600    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958602    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958605    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958608    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958610    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958613    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958615    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958618    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958627    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:30.248390 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958630    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958633    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958636    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.958642    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958755    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958760    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958762    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958765    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958769    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958773    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958776    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958779    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958783    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958786    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 00:40:30.248907 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958788    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958791    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958794    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958796    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958799    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958801    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958804    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958807    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958809    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958812    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958814    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958817    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958819    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958822    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958824    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958827    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958830    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958832    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958835    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958837    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 00:40:30.249349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958840    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958843    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958845    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958848    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958851    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958853    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958856    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958858    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958861    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958863    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958866    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958869    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958872    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958874    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958876    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958879    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958882    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958884    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958889    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958891    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 00:40:30.250000 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958894    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958897    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958899    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958903    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958906    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958909    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958911    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958914    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958917    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958920    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958922    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958925    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958927    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958930    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958932    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958935    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958938    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958940    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958943    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958945    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 00:40:30.250640 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958948    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958950    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958953    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958955    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958958    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958960    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958962    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958965    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958967    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958970    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958974    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958976    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958979    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958981    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958984    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958987    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958990    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958992    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958995    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.958997    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 00:40:30.251260 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959000    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959003    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959005    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959008    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959011    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959014    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959016    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959019    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:29.959021    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.959026    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.959175    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.961342    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.962666    2563 server.go:1019] "Starting client certificate rotation"
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.962764    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 18 00:40:30.251948 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.962800    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 18 00:40:30.252495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.965104    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 18 00:40:30.252495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.970551    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 18 00:40:30.252495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.980162    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 18 00:40:30.252495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.982975    2563 log.go:25] "Validated CRI v1 image API"
Jul 18 00:40:30.252495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.983970    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 18 00:40:30.252495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.986176    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c7c5dc4c-d220-48b5-8675-9d4a3c589959:/dev/nvme0n1p4 e624670b-1552-451a-a8d4-2aefb1cd29cc:/dev/nvme0n1p3]
Jul 18 00:40:30.252495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.986191    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.991473    2563 manager.go:216] Machine: {Timestamp:2026-07-18 00:40:29.990166513 +0000 UTC m=+0.310143887 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098351 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25628d5b187b79dcc216c6f0441d5e SystemUUID:ec25628d-5b18-7b79-dcc2-16c6f0441d5e BootID:26e5f2de-7211-4eb5-8f94-8b3dd61ce338 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6c:6e:90:61:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6c:6e:90:61:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:99:48:88:a1:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.991584    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.991671    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.993935    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.994400    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.994431    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.995816    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.995829    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.995839    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.996479    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:29.997969    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.002322    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.002477    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.002512    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.002524    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.002542    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 18 00:40:30.253297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.002554    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.002565    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.003003    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.003022    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.005463    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007027    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007077    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007753    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007766    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007772    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007778    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007786    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007791    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007799    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007804    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007811    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007816    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007825    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007830    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007853    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.007858    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.011463    2563 server.go:1268] "Started kubelet"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.012092    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.022492    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.022616    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.022864    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t8b58"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.023125    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.023821    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.024705    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.025160    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t8b58"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.029247    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.029322    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.029334    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-142.ec2.internal.18c33b70190bedf2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-142.ec2.internal,UID:ip-10-0-143-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-142.ec2.internal,},FirstTimestamp:2026-07-18 00:40:30.01143653 +0000 UTC m=+0.331413908,LastTimestamp:2026-07-18 00:40:30.01143653 +0000 UTC m=+0.331413908,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-142.ec2.internal,}"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.033728    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.033739    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.033805    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.034307    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.034428    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.034445    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.034554    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.034562    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 18 00:40:30.253850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.034730    2563 factory.go:55] Registering systemd factory
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.034747    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.035070    2563 factory.go:153] Registering CRI-O factory
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.035086    2563 factory.go:223] Registration of the crio container factory successfully
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.035142    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.035166    2563 factory.go:103] Registering Raw factory
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.035182    2563 manager.go:1196] Started watching for new ooms in manager
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.035587    2563 manager.go:349] Starting recovery of all containers
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.035947    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.037142    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.039848    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-142.ec2.internal\" not found" node="ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.045794    2563 manager.go:354] Recovery completed
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.049951    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.052095    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.052123    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.052133    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.052645    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.052651    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.052654    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.052670    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.053381    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.053404    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.053414    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.054838    2563 policy_none.go:49] "None policy: Start"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.054851    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.054860    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.056843    2563 policy_none.go:47] "Start"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:30.087636    2563 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/cpuset.cpus.effective": read /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/cpuset.cpus.effective: no such device
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.112792    2563 manager.go:337] "Starting Device Plugin manager"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.112825    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.112834    2563 server.go:87] "Starting device plugin registration server"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.113150    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.113162    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.113256    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.113357    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.113367    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.113847    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.113894    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.167706    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.168881    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.168920    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.168943    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.168982    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.170907    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.213853    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.214760    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.214789    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.214799    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:30.254932 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.214823    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.256262 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.225092    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.256262 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.225112    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-142.ec2.internal\": node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:30.256262 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.239983    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.270113    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"]
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.270192    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.271708    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.271735    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.271745    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.274120    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.274229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.274259    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.276939    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.276939    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.276986    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.276997    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.276969    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.277067    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.279315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.279338    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.280117    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.280147    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.280160    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeHasSufficientPID"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.307756    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-142.ec2.internal\" not found" node="ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.312547    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-142.ec2.internal\" not found" node="ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.339895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a137b0ed6b20ac3674d549e327e5ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"95a137b0ed6b20ac3674d549e327e5ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.339925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a137b0ed6b20ac3674d549e327e5ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"95a137b0ed6b20ac3674d549e327e5ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.339944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f41603fa42d7d0e66deed9946e4a9d78-config\") pod \"kube-apiserver-proxy-ip-10-0-143-142.ec2.internal\" (UID: \"f41603fa42d7d0e66deed9946e4a9d78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.340910    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.440353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a137b0ed6b20ac3674d549e327e5ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"95a137b0ed6b20ac3674d549e327e5ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.440392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a137b0ed6b20ac3674d549e327e5ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"95a137b0ed6b20ac3674d549e327e5ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.442891 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.440414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f41603fa42d7d0e66deed9946e4a9d78-config\") pod \"kube-apiserver-proxy-ip-10-0-143-142.ec2.internal\" (UID: \"f41603fa42d7d0e66deed9946e4a9d78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.443778 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.440465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f41603fa42d7d0e66deed9946e4a9d78-config\") pod \"kube-apiserver-proxy-ip-10-0-143-142.ec2.internal\" (UID: \"f41603fa42d7d0e66deed9946e4a9d78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.443778 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.440532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a137b0ed6b20ac3674d549e327e5ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"95a137b0ed6b20ac3674d549e327e5ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.443778 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.440564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a137b0ed6b20ac3674d549e327e5ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal\" (UID: \"95a137b0ed6b20ac3674d549e327e5ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.443778 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.442337    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:30.627155 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.543300    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:30.627155 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.614931    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.627155 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.620437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:30.820575 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.644234    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:30.820575 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.744863    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:31.031970 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.845551    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:31.031970 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:30.945645    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:31.031970 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.962767    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 18 00:40:31.031970 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.962943    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 18 00:40:31.031970 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:30.962974    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 18 00:40:31.031970 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.030937    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 00:35:30 +0000 UTC" deadline="2028-01-21 06:33:16.252421587 +0000 UTC"
Jul 18 00:40:31.031970 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.030973    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13253h52m45.221454047s"
Jul 18 00:40:31.183273 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:31.046663    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:31.183273 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.091794    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:31.183273 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:31.147131    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:31.318317 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:31.247672    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:31.430894 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:31.348266    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:31.430894 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:31.374618    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41603fa42d7d0e66deed9946e4a9d78.slice/crio-baa032dfd6235c17675cea9e2d409862f1334821f373ada80666f13864308654 WatchSource:0}: Error finding container baa032dfd6235c17675cea9e2d409862f1334821f373ada80666f13864308654: Status 404 returned error can't find the container with id baa032dfd6235c17675cea9e2d409862f1334821f373ada80666f13864308654
Jul 18 00:40:31.430894 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:31.374839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a137b0ed6b20ac3674d549e327e5ac.slice/crio-5d119720bb6f25bb221d94e60f3122f4900dd783cd1c5eab9a9b41f182ba317a WatchSource:0}: Error finding container 5d119720bb6f25bb221d94e60f3122f4900dd783cd1c5eab9a9b41f182ba317a: Status 404 returned error can't find the container with id 5d119720bb6f25bb221d94e60f3122f4900dd783cd1c5eab9a9b41f182ba317a
Jul 18 00:40:31.430894 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.378809    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 00:40:31.622566 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:31.448896    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-142.ec2.internal\" not found"
Jul 18 00:40:31.622566 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.539598    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:31.813669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.634579    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:31.813669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.649431    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 18 00:40:31.813669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.650424    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal"
Jul 18 00:40:31.813669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:31.661471    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 18 00:40:32.061064 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.005327    2563 apiserver.go:52] "Watching apiserver"
Jul 18 00:40:32.061064 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.035271    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.174747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal" event={"ID":"f41603fa42d7d0e66deed9946e4a9d78","Type":"ContainerStarted","Data":"baa032dfd6235c17675cea9e2d409862f1334821f373ada80666f13864308654"}
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.176046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal" event={"ID":"95a137b0ed6b20ac3674d549e327e5ac","Type":"ContainerStarted","Data":"5d119720bb6f25bb221d94e60f3122f4900dd783cd1c5eab9a9b41f182ba317a"}
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.194053    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.198862    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.199338    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-vv5qj","openshift-network-operator/iptables-alerter-gmdn2","openshift-ovn-kubernetes/ovnkube-node-clxs6","kube-system/global-pull-secret-syncer-g5fq8","kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2tk2x","openshift-image-registry/node-ca-tkd94","openshift-multus/multus-additional-cni-plugins-k4lcb","openshift-multus/network-metrics-daemon-9b8ml","kube-system/konnectivity-agent-ghhg9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7","openshift-dns/node-resolver-pgnlq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal","openshift-multus/multus-6zmpk"]
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.200443    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.200537    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdzvg"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.203393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.205779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.208202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.209236    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.209439    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.209657    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.209709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.209929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.209939    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.209969    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.209929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.210075    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7t2bp\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.210282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8hpx\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.210949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.211034    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.212708    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.212835    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.213265    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.213464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.213720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8466p\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.213870    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.214044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 18 00:40:32.224504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.214335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:32.226078 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.214405    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:32.226078 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.216101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.226078 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.218647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:32.226078 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.218738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:32.226078 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.221359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.226078 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.224199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.227159 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.226373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:32.229985 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.229102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.231699 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.231678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.239318 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.237974    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 18 00:40:32.242844 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.242764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 18 00:40:32.242974 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.242764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 18 00:40:32.243058 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.242999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 18 00:40:32.243109 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243087    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 18 00:40:32.243270 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 18 00:40:32.243570 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243435    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6sfkd\""
Jul 18 00:40:32.243570 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243490    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 18 00:40:32.243570 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243562    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 18 00:40:32.243762 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243638    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 18 00:40:32.243762 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243707    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xmj6x\""
Jul 18 00:40:32.243853 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 18 00:40:32.243853 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243782    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7vh6m\""
Jul 18 00:40:32.243956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243875    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xmvdr\""
Jul 18 00:40:32.243956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243945    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 18 00:40:32.244061 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.243948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 18 00:40:32.244061 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.244012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkjgn\""
Jul 18 00:40:32.244197 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.244070    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 18 00:40:32.244502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.244365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 18 00:40:32.244502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.244414    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6ll72\""
Jul 18 00:40:32.253484 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xpq4\" (UniqueName: \"kubernetes.io/projected/a92a03a0-14e8-4b1b-9358-9ac356022f35-kube-api-access-9xpq4\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.253624 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-lib-modules\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.253624 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-cni-binary-copy\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.253624 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-etc-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.253624 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysctl-d\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.253844 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-host\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.253844 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkw2h\" (UniqueName: \"kubernetes.io/projected/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-kube-api-access-gkw2h\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.253844 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-system-cni-dir\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.253844 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkr6\" (UniqueName: \"kubernetes.io/projected/ebf3259a-976a-4141-a2d9-456552d62d1a-kube-api-access-rzkr6\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.254038 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-iptables-alerter-script\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.254038 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/139e9ed7-f820-45ad-ac30-a24de9377afe-agent-certs\") pod \"konnectivity-agent-ghhg9\" (UID: \"139e9ed7-f820-45ad-ac30-a24de9377afe\") " pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:32.254038 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-socket-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.254038 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-system-cni-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.254038 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.254038 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.253976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-cni-netd\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.254038 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.254000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysctl-conf\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.254450 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.254045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-multus-certs\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.254450 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.254085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-cnibin\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.254450 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.254114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.254790 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.254762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.254889 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.254818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-ovnkube-config\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.254889 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.254845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:32.255244 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-socket-dir-parent\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.255413 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-kubelet\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.255413 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-etc-kubernetes\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.255413 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-ovnkube-script-lib\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.255413 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-host-slash\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.255413 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-device-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c17a371-633b-4139-a838-ffa1e1902d90-tmp\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-netns\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-run-netns\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-daemon-config\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-node-log\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-cni-bin\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-dbus\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.255732 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-run\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-cni-binary-copy\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beeaa768-6025-4191-9306-1c75abdd1477-serviceca\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-slash\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-ovn\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-sys-fs\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.255987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-systemd\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-cnibin\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-registration-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-kubernetes\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-sys\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9pqx\" (UniqueName: \"kubernetes.io/projected/3c17a371-633b-4139-a838-ffa1e1902d90-kube-api-access-s9pqx\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-systemd\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.256300 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-cni-bin\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c200b84-577b-45d8-a6b6-e715ce2d4262-hosts-file\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c200b84-577b-45d8-a6b6-e715ce2d4262-tmp-dir\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-os-release\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-env-overrides\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c17a371-633b-4139-a838-ffa1e1902d90-etc-tuned\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-kubelet-config\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-modprobe-d\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w767h\" (UniqueName: \"kubernetes.io/projected/75ee96b7-bcc7-4f75-bf11-52a674359b51-kube-api-access-w767h\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-kubelet\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-log-socket\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4wc\" (UniqueName: \"kubernetes.io/projected/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-kube-api-access-xq4wc\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-cni-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljk9d\" (UniqueName: \"kubernetes.io/projected/6c200b84-577b-45d8-a6b6-e715ce2d4262-kube-api-access-ljk9d\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.257096 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6fd\" (UniqueName: \"kubernetes.io/projected/02374191-e591-4213-b783-3fc5d47e9625-kube-api-access-8r6fd\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-systemd-units\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-var-lib-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/139e9ed7-f820-45ad-ac30-a24de9377afe-konnectivity-ca\") pod \"konnectivity-agent-ghhg9\" (UID: \"139e9ed7-f820-45ad-ac30-a24de9377afe\") " pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-hostroot\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebf3259a-976a-4141-a2d9-456552d62d1a-ovn-node-metrics-cert\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysconfig\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.256984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-cni-multus\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.257010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-conf-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.257037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncb8h\" (UniqueName: \"kubernetes.io/projected/beeaa768-6025-4191-9306-1c75abdd1477-kube-api-access-ncb8h\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.257062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-os-release\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.257716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.257104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-k8s-cni-cncf-io\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.258152 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.257702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beeaa768-6025-4191-9306-1c75abdd1477-host\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.259611 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.259585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-var-lib-kubelet\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.360595 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysctl-conf\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.360595 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-multus-certs\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.360595 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-cnibin\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.360595 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.360595 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-ovnkube-config\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-socket-dir-parent\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-kubelet\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-etc-kubernetes\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-ovnkube-script-lib\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-run-ovn-kubernetes\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-host-slash\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-host-slash\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-device-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.360986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-multus-certs\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.361647 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-cnibin\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.361647 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-ovnkube-config\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.361647 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c17a371-633b-4139-a838-ffa1e1902d90-tmp\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-netns\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-device-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-run-netns\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-run-netns\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.360701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysctl-conf\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-daemon-config\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-etc-kubernetes\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.361807 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-node-log\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-cni-bin\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-dbus\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-run\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-cni-binary-copy\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beeaa768-6025-4191-9306-1c75abdd1477-serviceca\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-slash\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-ovn\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.361995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-sys-fs\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362046    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-systemd\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-cnibin\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-registration-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-kubernetes\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-sys\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9pqx\" (UniqueName: \"kubernetes.io/projected/3c17a371-633b-4139-a838-ffa1e1902d90-kube-api-access-s9pqx\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.362325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-systemd\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-ovnkube-script-lib\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-cni-bin\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-netns\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c200b84-577b-45d8-a6b6-e715ce2d4262-hosts-file\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c200b84-577b-45d8-a6b6-e715ce2d4262-tmp-dir\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-os-release\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-kubernetes\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-env-overrides\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c17a371-633b-4139-a838-ffa1e1902d90-etc-tuned\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-kubelet\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-sys-fs\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-sys\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-kubelet-config\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-modprobe-d\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w767h\" (UniqueName: \"kubernetes.io/projected/75ee96b7-bcc7-4f75-bf11-52a674359b51-kube-api-access-w767h\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-kubelet\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.363296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-log-socket\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-systemd\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4wc\" (UniqueName: \"kubernetes.io/projected/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-kube-api-access-xq4wc\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.362861    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.362918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret podName:2a5b6fba-2c56-4526-9e0f-21ca35cba6f2 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:32.862902116 +0000 UTC m=+3.182879479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret") pod "global-pull-secret-syncer-g5fq8" (UID: "2a5b6fba-2c56-4526-9e0f-21ca35cba6f2") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-systemd\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-cni-bin\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-cni-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-run\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-kubelet\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c200b84-577b-45d8-a6b6-e715ce2d4262-hosts-file\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-socket-dir-parent\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-slash\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-log-socket\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-kubelet-config\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-daemon-config\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-modprobe-d\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-ovn\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c200b84-577b-45d8-a6b6-e715ce2d4262-tmp-dir\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.363442    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-os-release\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.363494    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs podName:75ee96b7-bcc7-4f75-bf11-52a674359b51 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:32.863477805 +0000 UTC m=+3.183455178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs") pod "network-metrics-daemon-9b8ml" (UID: "75ee96b7-bcc7-4f75-bf11-52a674359b51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-cnibin\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.362772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-cni-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-cni-bin\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljk9d\" (UniqueName: \"kubernetes.io/projected/6c200b84-577b-45d8-a6b6-e715ce2d4262-kube-api-access-ljk9d\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6fd\" (UniqueName: \"kubernetes.io/projected/02374191-e591-4213-b783-3fc5d47e9625-kube-api-access-8r6fd\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-dbus\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebf3259a-976a-4141-a2d9-456552d62d1a-env-overrides\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-node-log\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-systemd-units\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-systemd-units\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-registration-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-run-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-cni-binary-copy\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-var-lib-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-var-lib-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/139e9ed7-f820-45ad-ac30-a24de9377afe-konnectivity-ca\") pod \"konnectivity-agent-ghhg9\" (UID: \"139e9ed7-f820-45ad-ac30-a24de9377afe\") " pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-hostroot\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebf3259a-976a-4141-a2d9-456552d62d1a-ovn-node-metrics-cert\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysconfig\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-hostroot\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-cni-multus\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-conf-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-var-lib-cni-multus\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncb8h\" (UniqueName: \"kubernetes.io/projected/beeaa768-6025-4191-9306-1c75abdd1477-kube-api-access-ncb8h\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-os-release\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.365728 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-k8s-cni-cncf-io\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beeaa768-6025-4191-9306-1c75abdd1477-host\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/beeaa768-6025-4191-9306-1c75abdd1477-serviceca\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-var-lib-kubelet\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xpq4\" (UniqueName: \"kubernetes.io/projected/a92a03a0-14e8-4b1b-9358-9ac356022f35-kube-api-access-9xpq4\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-lib-modules\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-cni-binary-copy\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-etc-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysctl-d\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/139e9ed7-f820-45ad-ac30-a24de9377afe-konnectivity-ca\") pod \"konnectivity-agent-ghhg9\" (UID: \"139e9ed7-f820-45ad-ac30-a24de9377afe\") " pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-host-run-k8s-cni-cncf-io\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-host\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-os-release\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkw2h\" (UniqueName: \"kubernetes.io/projected/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-kube-api-access-gkw2h\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/beeaa768-6025-4191-9306-1c75abdd1477-host\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-system-cni-dir\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkr6\" (UniqueName: \"kubernetes.io/projected/ebf3259a-976a-4141-a2d9-456552d62d1a-kube-api-access-rzkr6\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-host\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.366619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-multus-conf-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.364169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-var-lib-kubelet\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.363946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysconfig\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.365472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-iptables-alerter-script\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.365532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/139e9ed7-f820-45ad-ac30-a24de9377afe-agent-certs\") pod \"konnectivity-agent-ghhg9\" (UID: \"139e9ed7-f820-45ad-ac30-a24de9377afe\") " pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.365562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-socket-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.365598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-system-cni-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.365634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.365669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-cni-netd\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.365809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-host-cni-netd\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.366605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-iptables-alerter-script\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.367045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebf3259a-976a-4141-a2d9-456552d62d1a-etc-openvswitch\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.367529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.367188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-etc-sysctl-d\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.368087 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.367680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02374191-e591-4213-b783-3fc5d47e9625-cni-binary-copy\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.368087 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.367718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-system-cni-dir\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.368087 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.367804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c17a371-633b-4139-a838-ffa1e1902d90-lib-modules\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.368675 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.368466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c17a371-633b-4139-a838-ffa1e1902d90-tmp\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.368675 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.368561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a92a03a0-14e8-4b1b-9358-9ac356022f35-socket-dir\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.368675 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.368641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02374191-e591-4213-b783-3fc5d47e9625-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.368675 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.368645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-system-cni-dir\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.371581 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.368782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebf3259a-976a-4141-a2d9-456552d62d1a-ovn-node-metrics-cert\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.371581 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.369071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c17a371-633b-4139-a838-ffa1e1902d90-etc-tuned\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.372054 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.372021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/139e9ed7-f820-45ad-ac30-a24de9377afe-agent-certs\") pod \"konnectivity-agent-ghhg9\" (UID: \"139e9ed7-f820-45ad-ac30-a24de9377afe\") " pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:32.373306 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.373260    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:32.373306 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.373289    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:32.373306 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.373304    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tt5cb for pod openshift-network-diagnostics/network-check-target-vv5qj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:32.373602 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.373451    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb podName:509a8c3e-3258-4abb-83a6-ad5bb4722523 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:32.873429172 +0000 UTC m=+3.193406555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tt5cb" (UniqueName: "kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb") pod "network-check-target-vv5qj" (UID: "509a8c3e-3258-4abb-83a6-ad5bb4722523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:32.375931 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.375902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9pqx\" (UniqueName: \"kubernetes.io/projected/3c17a371-633b-4139-a838-ffa1e1902d90-kube-api-access-s9pqx\") pod \"tuned-2tk2x\" (UID: \"3c17a371-633b-4139-a838-ffa1e1902d90\") " pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.376658 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.376590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljk9d\" (UniqueName: \"kubernetes.io/projected/6c200b84-577b-45d8-a6b6-e715ce2d4262-kube-api-access-ljk9d\") pod \"node-resolver-pgnlq\" (UID: \"6c200b84-577b-45d8-a6b6-e715ce2d4262\") " pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.379235 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.379210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkw2h\" (UniqueName: \"kubernetes.io/projected/c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e-kube-api-access-gkw2h\") pod \"multus-6zmpk\" (UID: \"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e\") " pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.379453 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.379308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w767h\" (UniqueName: \"kubernetes.io/projected/75ee96b7-bcc7-4f75-bf11-52a674359b51-kube-api-access-w767h\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:32.379673 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.379647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xpq4\" (UniqueName: \"kubernetes.io/projected/a92a03a0-14e8-4b1b-9358-9ac356022f35-kube-api-access-9xpq4\") pod \"aws-ebs-csi-driver-node-ldbr7\" (UID: \"a92a03a0-14e8-4b1b-9358-9ac356022f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.380787 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.380750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkr6\" (UniqueName: \"kubernetes.io/projected/ebf3259a-976a-4141-a2d9-456552d62d1a-kube-api-access-rzkr6\") pod \"ovnkube-node-clxs6\" (UID: \"ebf3259a-976a-4141-a2d9-456552d62d1a\") " pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.381052 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.381030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4wc\" (UniqueName: \"kubernetes.io/projected/dd6e4dad-7434-445e-a8d1-38ecdb6124f8-kube-api-access-xq4wc\") pod \"iptables-alerter-gmdn2\" (UID: \"dd6e4dad-7434-445e-a8d1-38ecdb6124f8\") " pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.381174 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.381111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncb8h\" (UniqueName: \"kubernetes.io/projected/beeaa768-6025-4191-9306-1c75abdd1477-kube-api-access-ncb8h\") pod \"node-ca-tkd94\" (UID: \"beeaa768-6025-4191-9306-1c75abdd1477\") " pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.381987 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.381964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6fd\" (UniqueName: \"kubernetes.io/projected/02374191-e591-4213-b783-3fc5d47e9625-kube-api-access-8r6fd\") pod \"multus-additional-cni-plugins-k4lcb\" (UID: \"02374191-e591-4213-b783-3fc5d47e9625\") " pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.515325 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.515245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tkd94"
Jul 18 00:40:32.525101 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.525069    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k4lcb"
Jul 18 00:40:32.532306 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.532285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:32.540418 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.540393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tk2x"
Jul 18 00:40:32.541553 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.541535    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 00:40:32.548961 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.548939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6zmpk"
Jul 18 00:40:32.567134 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.567102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmdn2"
Jul 18 00:40:32.576226 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.576203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:32.585048 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.585026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7"
Jul 18 00:40:32.590934 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.590911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgnlq"
Jul 18 00:40:32.869757 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.869663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:32.869757 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.869721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:32.869977 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.869820    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:32.869977 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.869886    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret podName:2a5b6fba-2c56-4526-9e0f-21ca35cba6f2 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:33.869871157 +0000 UTC m=+4.189848519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret") pod "global-pull-secret-syncer-g5fq8" (UID: "2a5b6fba-2c56-4526-9e0f-21ca35cba6f2") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:32.869977 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.869821    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:32.870184 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.869980    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs podName:75ee96b7-bcc7-4f75-bf11-52a674359b51 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:33.869959812 +0000 UTC m=+4.189937192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs") pod "network-metrics-daemon-9b8ml" (UID: "75ee96b7-bcc7-4f75-bf11-52a674359b51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:32.970416 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:32.970371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:32.970593 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.970554    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:32.970593 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.970579    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:32.970593 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.970590    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tt5cb for pod openshift-network-diagnostics/network-check-target-vv5qj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:32.970741 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:32.970659    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb podName:509a8c3e-3258-4abb-83a6-ad5bb4722523 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:33.970639021 +0000 UTC m=+4.290616389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tt5cb" (UniqueName: "kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb") pod "network-check-target-vv5qj" (UID: "509a8c3e-3258-4abb-83a6-ad5bb4722523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:33.213821 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:33.213691    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 00:35:32 +0000 UTC" deadline="2028-03-30 08:35:20.956983612 +0000 UTC"
Jul 18 00:40:33.213821 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:33.213756    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14911h54m47.743231912s"
Jul 18 00:40:33.504773 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:33.504741    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf3259a_976a_4141_a2d9_456552d62d1a.slice/crio-6b40a07c3d16243af55292406da1cf110be23c4a1c2b6277aa3f2291444618e3 WatchSource:0}: Error finding container 6b40a07c3d16243af55292406da1cf110be23c4a1c2b6277aa3f2291444618e3: Status 404 returned error can't find the container with id 6b40a07c3d16243af55292406da1cf110be23c4a1c2b6277aa3f2291444618e3
Jul 18 00:40:33.505433 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:33.505404    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92a03a0_14e8_4b1b_9358_9ac356022f35.slice/crio-be1e4088afb25cf0ac1270c7a1ff04cddda1609b28c562b001e74893c4334b38 WatchSource:0}: Error finding container be1e4088afb25cf0ac1270c7a1ff04cddda1609b28c562b001e74893c4334b38: Status 404 returned error can't find the container with id be1e4088afb25cf0ac1270c7a1ff04cddda1609b28c562b001e74893c4334b38
Jul 18 00:40:33.506294 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:33.506264    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c200b84_577b_45d8_a6b6_e715ce2d4262.slice/crio-5523128f43f35d30d91d4650017f42e65f7efafb877869b0b070fe3e40bbc4b4 WatchSource:0}: Error finding container 5523128f43f35d30d91d4650017f42e65f7efafb877869b0b070fe3e40bbc4b4: Status 404 returned error can't find the container with id 5523128f43f35d30d91d4650017f42e65f7efafb877869b0b070fe3e40bbc4b4
Jul 18 00:40:33.507022 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:33.506981    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6e4dad_7434_445e_a8d1_38ecdb6124f8.slice/crio-dce6022566ad2c0853621c1fb9ddc89b601916c398de0736e40d90e248413d74 WatchSource:0}: Error finding container dce6022566ad2c0853621c1fb9ddc89b601916c398de0736e40d90e248413d74: Status 404 returned error can't find the container with id dce6022566ad2c0853621c1fb9ddc89b601916c398de0736e40d90e248413d74
Jul 18 00:40:33.513012 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:33.512982    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139e9ed7_f820_45ad_ac30_a24de9377afe.slice/crio-a38907e45a99429f25c4f48c9d8341be3d84f86c3eba6b4915b0704bd820fa5f WatchSource:0}: Error finding container a38907e45a99429f25c4f48c9d8341be3d84f86c3eba6b4915b0704bd820fa5f: Status 404 returned error can't find the container with id a38907e45a99429f25c4f48c9d8341be3d84f86c3eba6b4915b0704bd820fa5f
Jul 18 00:40:33.513463 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:33.513408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeeaa768_6025_4191_9306_1c75abdd1477.slice/crio-d2a484c9338a487583bc9c2d443e9bcd0d266bd5c4ba96d9685e66642e881b4c WatchSource:0}: Error finding container d2a484c9338a487583bc9c2d443e9bcd0d266bd5c4ba96d9685e66642e881b4c: Status 404 returned error can't find the container with id d2a484c9338a487583bc9c2d443e9bcd0d266bd5c4ba96d9685e66642e881b4c
Jul 18 00:40:33.514700 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:40:33.514633    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c17a371_633b_4139_a838_ffa1e1902d90.slice/crio-ce6a13c11e82ad8ac5da2e26ff7eb2c740119ceadd7c93c924efb43e61d8b1ea WatchSource:0}: Error finding container ce6a13c11e82ad8ac5da2e26ff7eb2c740119ceadd7c93c924efb43e61d8b1ea: Status 404 returned error can't find the container with id ce6a13c11e82ad8ac5da2e26ff7eb2c740119ceadd7c93c924efb43e61d8b1ea
Jul 18 00:40:33.878904 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:33.878668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:33.879064 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:33.878953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:33.879064 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:33.878849    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:33.879064 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:33.879045    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret podName:2a5b6fba-2c56-4526-9e0f-21ca35cba6f2 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:35.879019685 +0000 UTC m=+6.198997050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret") pod "global-pull-secret-syncer-g5fq8" (UID: "2a5b6fba-2c56-4526-9e0f-21ca35cba6f2") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:33.879064 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:33.879066    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:33.879368 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:33.879135    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs podName:75ee96b7-bcc7-4f75-bf11-52a674359b51 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:35.879122072 +0000 UTC m=+6.199099449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs") pod "network-metrics-daemon-9b8ml" (UID: "75ee96b7-bcc7-4f75-bf11-52a674359b51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:33.980012 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:33.979971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:33.980732 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:33.980235    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:33.980732 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:33.980261    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:33.980732 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:33.980275    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tt5cb for pod openshift-network-diagnostics/network-check-target-vv5qj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:33.980732 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:33.980342    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb podName:509a8c3e-3258-4abb-83a6-ad5bb4722523 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:35.980320832 +0000 UTC m=+6.300298213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tt5cb" (UniqueName: "kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb") pod "network-check-target-vv5qj" (UID: "509a8c3e-3258-4abb-83a6-ad5bb4722523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:34.171664 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.171015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:34.171664 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:34.171182    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:34.171664 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.171197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:34.171664 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.171019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:34.171664 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:34.171340    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:34.171664 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:34.171429    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:34.188295 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.188225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tk2x" event={"ID":"3c17a371-633b-4139-a838-ffa1e1902d90","Type":"ContainerStarted","Data":"ce6a13c11e82ad8ac5da2e26ff7eb2c740119ceadd7c93c924efb43e61d8b1ea"}
Jul 18 00:40:34.194567 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.194490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tkd94" event={"ID":"beeaa768-6025-4191-9306-1c75abdd1477","Type":"ContainerStarted","Data":"d2a484c9338a487583bc9c2d443e9bcd0d266bd5c4ba96d9685e66642e881b4c"}
Jul 18 00:40:34.212219 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.212182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmdn2" event={"ID":"dd6e4dad-7434-445e-a8d1-38ecdb6124f8","Type":"ContainerStarted","Data":"dce6022566ad2c0853621c1fb9ddc89b601916c398de0736e40d90e248413d74"}
Jul 18 00:40:34.214034 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.213975    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 00:35:32 +0000 UTC" deadline="2028-01-20 02:04:05.125677922 +0000 UTC"
Jul 18 00:40:34.214034 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.214034    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13225h23m30.91164861s"
Jul 18 00:40:34.222163 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.222072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgnlq" event={"ID":"6c200b84-577b-45d8-a6b6-e715ce2d4262","Type":"ContainerStarted","Data":"5523128f43f35d30d91d4650017f42e65f7efafb877869b0b070fe3e40bbc4b4"}
Jul 18 00:40:34.232166 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.231938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7" event={"ID":"a92a03a0-14e8-4b1b-9358-9ac356022f35","Type":"ContainerStarted","Data":"be1e4088afb25cf0ac1270c7a1ff04cddda1609b28c562b001e74893c4334b38"}
Jul 18 00:40:34.246009 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.245972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6zmpk" event={"ID":"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e","Type":"ContainerStarted","Data":"6cb07cde41331eab0c5ea63624a1e893e2a0ca963f2b98baa1cfd5c2ace8650a"}
Jul 18 00:40:34.258982 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.258946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghhg9" event={"ID":"139e9ed7-f820-45ad-ac30-a24de9377afe","Type":"ContainerStarted","Data":"a38907e45a99429f25c4f48c9d8341be3d84f86c3eba6b4915b0704bd820fa5f"}
Jul 18 00:40:34.267483 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.267448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"6b40a07c3d16243af55292406da1cf110be23c4a1c2b6277aa3f2291444618e3"}
Jul 18 00:40:34.295964 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.295923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal" event={"ID":"f41603fa42d7d0e66deed9946e4a9d78","Type":"ContainerStarted","Data":"9840157bc241afb604cc21f5775c532e2c0164886a72721d22d0e0fea4043cbd"}
Jul 18 00:40:34.300475 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.300440    2563 generic.go:356] "Generic (PLEG): container finished" podID="95a137b0ed6b20ac3674d549e327e5ac" containerID="1732bed9678a467954af7153ff1de053d54e76c6ae683cdf8b6ee431a65e232f" exitCode=0
Jul 18 00:40:34.300788 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.300718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal" event={"ID":"95a137b0ed6b20ac3674d549e327e5ac","Type":"ContainerDied","Data":"1732bed9678a467954af7153ff1de053d54e76c6ae683cdf8b6ee431a65e232f"}
Jul 18 00:40:34.312235 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.312183    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-142.ec2.internal" podStartSLOduration=2.312162312 podStartE2EDuration="2.312162312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:34.311375812 +0000 UTC m=+4.631353200" watchObservedRunningTime="2026-07-18 00:40:34.312162312 +0000 UTC m=+4.632139700"
Jul 18 00:40:34.332045 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:34.331961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" event={"ID":"02374191-e591-4213-b783-3fc5d47e9625","Type":"ContainerStarted","Data":"a303db0e12ccdf39535cba0d31ae571b8e33ce4407a57e9c6d146f97558f12a1"}
Jul 18 00:40:35.348299 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:35.348215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal" event={"ID":"95a137b0ed6b20ac3674d549e327e5ac","Type":"ContainerStarted","Data":"b14eb4e6963d07d718dc2a1281b147a64b65329da3ae62903b05119de4b976e3"}
Jul 18 00:40:35.898109 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:35.897277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:35.898109 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:35.897365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:35.898109 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:35.897529    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:35.898109 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:35.897596    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs podName:75ee96b7-bcc7-4f75-bf11-52a674359b51 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:39.897575828 +0000 UTC m=+10.217553198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs") pod "network-metrics-daemon-9b8ml" (UID: "75ee96b7-bcc7-4f75-bf11-52a674359b51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:35.898109 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:35.898014    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:35.898109 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:35.898070    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret podName:2a5b6fba-2c56-4526-9e0f-21ca35cba6f2 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:39.89804831 +0000 UTC m=+10.218025679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret") pod "global-pull-secret-syncer-g5fq8" (UID: "2a5b6fba-2c56-4526-9e0f-21ca35cba6f2") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:35.998522 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:35.997762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:35.998522 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:35.997981    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:35.998522 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:35.998005    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:35.998522 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:35.998019    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tt5cb for pod openshift-network-diagnostics/network-check-target-vv5qj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:35.998522 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:35.998082    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb podName:509a8c3e-3258-4abb-83a6-ad5bb4722523 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:39.998062377 +0000 UTC m=+10.318039741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tt5cb" (UniqueName: "kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb") pod "network-check-target-vv5qj" (UID: "509a8c3e-3258-4abb-83a6-ad5bb4722523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:36.170066 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:36.169499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:36.170066 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:36.169502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:36.170066 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:36.169646    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:36.170066 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:36.169673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:36.170066 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:36.169813    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:36.170066 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:36.169927    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:38.171170 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:38.169240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:38.171170 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:38.169271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:38.171170 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:38.169383    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:38.171170 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:38.169452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:38.171170 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:38.169592    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:38.171170 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:38.169702    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:39.933302 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:39.933242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:39.933802 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:39.933337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:39.933802 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:39.933438    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:39.933802 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:39.933478    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:39.933802 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:39.933523    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret podName:2a5b6fba-2c56-4526-9e0f-21ca35cba6f2 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:47.933500592 +0000 UTC m=+18.253477958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret") pod "global-pull-secret-syncer-g5fq8" (UID: "2a5b6fba-2c56-4526-9e0f-21ca35cba6f2") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:39.933802 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:39.933547    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs podName:75ee96b7-bcc7-4f75-bf11-52a674359b51 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:47.933535284 +0000 UTC m=+18.253512650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs") pod "network-metrics-daemon-9b8ml" (UID: "75ee96b7-bcc7-4f75-bf11-52a674359b51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:40.035216 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:40.034683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:40.035216 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:40.034908    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:40.035216 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:40.034987    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:40.035216 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:40.035003    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tt5cb for pod openshift-network-diagnostics/network-check-target-vv5qj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:40.035216 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:40.035075    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb podName:509a8c3e-3258-4abb-83a6-ad5bb4722523 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:48.035051909 +0000 UTC m=+18.355029297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tt5cb" (UniqueName: "kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb") pod "network-check-target-vv5qj" (UID: "509a8c3e-3258-4abb-83a6-ad5bb4722523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:40.171126 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:40.170569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:40.171126 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:40.170705    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:40.171126 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:40.170912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:40.171126 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:40.171003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:40.171126 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:40.171092    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:40.171126 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:40.171092    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:42.170868 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:42.170829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:42.171369 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:42.170981    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:42.171369 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:42.171299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:42.171508 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:42.171450    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:42.171567 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:42.171504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:42.171615 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:42.171597    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:44.169217 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:44.169182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:44.169684 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:44.169225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:44.169684 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:44.169182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:44.169684 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:44.169306    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:44.169684 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:44.169358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:44.169684 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:44.169425    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:46.169817 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:46.169780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:46.170254 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:46.169789    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:46.170254 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:46.169928    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:46.170254 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:46.169946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:46.170254 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:46.170032    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:46.170254 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:46.170138    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:47.998465 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:47.998424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:47.998951 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:47.998487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:47.998951 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:47.998613    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:47.998951 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:47.998667    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:47.998951 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:47.998702    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret podName:2a5b6fba-2c56-4526-9e0f-21ca35cba6f2 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:03.998679132 +0000 UTC m=+34.318656510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret") pod "global-pull-secret-syncer-g5fq8" (UID: "2a5b6fba-2c56-4526-9e0f-21ca35cba6f2") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:40:47.998951 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:47.998724    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs podName:75ee96b7-bcc7-4f75-bf11-52a674359b51 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:03.998712489 +0000 UTC m=+34.318689869 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs") pod "network-metrics-daemon-9b8ml" (UID: "75ee96b7-bcc7-4f75-bf11-52a674359b51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:40:48.099482 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:48.099418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:48.099653 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:48.099626    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:40:48.099714 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:48.099655    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:40:48.099714 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:48.099666    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tt5cb for pod openshift-network-diagnostics/network-check-target-vv5qj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:48.099803 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:48.099720    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb podName:509a8c3e-3258-4abb-83a6-ad5bb4722523 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:04.099703368 +0000 UTC m=+34.419680729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tt5cb" (UniqueName: "kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb") pod "network-check-target-vv5qj" (UID: "509a8c3e-3258-4abb-83a6-ad5bb4722523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:40:48.170176 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:48.170139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:48.170176 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:48.170180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:48.170410 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:48.170147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:48.170410 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:48.170275    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:48.170410 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:48.170355    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:48.170553 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:48.170445    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:50.170857 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:50.170820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:50.171368 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:50.170926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:50.171368 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:50.170941    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:50.171368 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:50.171077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:50.171368 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:50.171079    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:50.171368 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:50.171176    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:52.170202 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:52.170161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:52.170651 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:52.170326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:52.170651 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:52.170337    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:52.170651 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:52.170403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:52.170651 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:52.170518    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:52.170651 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:52.170593    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:53.389281 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.388976    2563 generic.go:356] "Generic (PLEG): container finished" podID="02374191-e591-4213-b783-3fc5d47e9625" containerID="17774ab8a62f459738699d74c021d817d5f965e50f8639e592ee298fe19773af" exitCode=0
Jul 18 00:40:53.390130 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.389084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" event={"ID":"02374191-e591-4213-b783-3fc5d47e9625","Type":"ContainerDied","Data":"17774ab8a62f459738699d74c021d817d5f965e50f8639e592ee298fe19773af"}
Jul 18 00:40:53.390829 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.390803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tk2x" event={"ID":"3c17a371-633b-4139-a838-ffa1e1902d90","Type":"ContainerStarted","Data":"294cdcc69ace8cae057016cb6e651107e5f854e972deb85355c90ceb5e292722"}
Jul 18 00:40:53.392247 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.392227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tkd94" event={"ID":"beeaa768-6025-4191-9306-1c75abdd1477","Type":"ContainerStarted","Data":"423081f73606aa7037aacea8a8bf0873f227ca999fc0d043a20d87a0288d15bf"}
Jul 18 00:40:53.393652 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.393630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgnlq" event={"ID":"6c200b84-577b-45d8-a6b6-e715ce2d4262","Type":"ContainerStarted","Data":"a944151f323222383fd7a96b383831b2bd1a02eaea0afde2ffb5638e2de76d49"}
Jul 18 00:40:53.395126 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.395001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7" event={"ID":"a92a03a0-14e8-4b1b-9358-9ac356022f35","Type":"ContainerStarted","Data":"7ce74c1d599ad8815abeff03ac627895ca1804872b2ce2b1cdac6889e82a9840"}
Jul 18 00:40:53.396446 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.396425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6zmpk" event={"ID":"c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e","Type":"ContainerStarted","Data":"c1424ec80c35f5c15dd647cac91be57db99d9cd258c6c0ecab0576ef677a94af"}
Jul 18 00:40:53.397650 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.397631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghhg9" event={"ID":"139e9ed7-f820-45ad-ac30-a24de9377afe","Type":"ContainerStarted","Data":"9e4e05c8d643aae36a209b861f3638d4c6c3c912ab5b5f736dfecf2250dd5b1b"}
Jul 18 00:40:53.399982 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.399962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"0079984b1dc088a24bb1593c3917069a1067ea6f63ad60ae8fea65e07a097893"}
Jul 18 00:40:53.400098 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.400034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"4b1d43fe3aec1836f8076f5430781b91e6aaf0c50a2410039f690976a1f2799a"}
Jul 18 00:40:53.400166 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.400112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"6b1b17259c303516c6cdeec580099a0e0b4af7601d0770ae44100ab475efbac3"}
Jul 18 00:40:53.400226 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.400163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"c42ddadc2bd3431887ddba3dfb7a3105a6be628f4b1a70cd48dc8368f41aa2d3"}
Jul 18 00:40:53.400226 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.400213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"e96af689a2ffeba777e571285e9adc931c7602afd20030704a90829aa30a485d"}
Jul 18 00:40:53.410775 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.410725    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-142.ec2.internal" podStartSLOduration=21.410713976 podStartE2EDuration="21.410713976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:35.366054878 +0000 UTC m=+5.686032267" watchObservedRunningTime="2026-07-18 00:40:53.410713976 +0000 UTC m=+23.730691358"
Jul 18 00:40:53.437273 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.437213    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pgnlq" podStartSLOduration=6.8128274300000005 podStartE2EDuration="22.437193706s" totalImagesPullingTime="15.624366276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:53.436761692 +0000 UTC m=+23.756739077" watchObservedRunningTime="2026-07-18 00:40:53.437193706 +0000 UTC m=+23.757171093"
Jul 18 00:40:53.438146 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.438104    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ghhg9" podStartSLOduration=6.819031754 podStartE2EDuration="22.438092939s" totalImagesPullingTime="15.619061185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:53.423454386 +0000 UTC m=+23.743431773" watchObservedRunningTime="2026-07-18 00:40:53.438092939 +0000 UTC m=+23.758070325"
Jul 18 00:40:53.464115 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.464042    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2tk2x" podStartSLOduration=3.145295676 podStartE2EDuration="22.46402324s" totalImagesPullingTime="19.318727564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:53.463785563 +0000 UTC m=+23.783762949" watchObservedRunningTime="2026-07-18 00:40:53.46402324 +0000 UTC m=+23.784000625"
Jul 18 00:40:53.464292 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.464203    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tkd94" podStartSLOduration=3.161818577 podStartE2EDuration="22.464197484s" totalImagesPullingTime="19.302378907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:53.449660648 +0000 UTC m=+23.769638032" watchObservedRunningTime="2026-07-18 00:40:53.464197484 +0000 UTC m=+23.784174872"
Jul 18 00:40:53.943611 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:53.943579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:54.169709 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.169662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:54.169709 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.169693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:54.169911 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.169662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:54.169911 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:54.169800    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:54.170090 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:54.169948    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:54.170090 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:54.170032    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:54.271109 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.271085    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 18 00:40:54.404603 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.404507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7" event={"ID":"a92a03a0-14e8-4b1b-9358-9ac356022f35","Type":"ContainerStarted","Data":"08fce88dd3d9e916ebdc0faeddd946e6c16c36c75c752626d8f60e8882ebceb9"}
Jul 18 00:40:54.407804 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.407773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"7607cf73f79b815a93dd45405223cec53ebfa10e482baa24c19b88710deafa3f"}
Jul 18 00:40:54.409566 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.409518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmdn2" event={"ID":"dd6e4dad-7434-445e-a8d1-38ecdb6124f8","Type":"ContainerStarted","Data":"b18178e183aa8ca0b7f343fd076d081bde222aa36399b26f6fa4c3efb95c53bf"}
Jul 18 00:40:54.424510 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.424423    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6zmpk" podStartSLOduration=4.06688537 podStartE2EDuration="23.424407528s" totalImagesPullingTime="19.357522158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:53.481814394 +0000 UTC m=+23.801791778" watchObservedRunningTime="2026-07-18 00:40:54.424407528 +0000 UTC m=+24.744384961"
Jul 18 00:40:54.424664 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:54.424635    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gmdn2" podStartSLOduration=4.110908071 podStartE2EDuration="23.424620888s" totalImagesPullingTime="19.313712817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:54.424125083 +0000 UTC m=+24.744102467" watchObservedRunningTime="2026-07-18 00:40:54.424620888 +0000 UTC m=+24.744598271"
Jul 18 00:40:55.131232 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:55.131109    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T00:40:54.271106569Z","UUID":"5522322f-cc6c-4346-8699-0e199abd585d","Handler":null,"Name":"","Endpoint":""}
Jul 18 00:40:55.132725 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:55.132698    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 18 00:40:55.132867 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:55.132743    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 18 00:40:55.414523 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:55.414437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7" event={"ID":"a92a03a0-14e8-4b1b-9358-9ac356022f35","Type":"ContainerStarted","Data":"de7423bfd885fd2188b120710d61aa7c41840fc97f2f627b197c7efefaa4207c"}
Jul 18 00:40:55.432872 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:55.432821    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldbr7" podStartSLOduration=3.016996892 podStartE2EDuration="24.432805373s" totalImagesPullingTime="21.415808481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:55.432623961 +0000 UTC m=+25.752601345" watchObservedRunningTime="2026-07-18 00:40:55.432805373 +0000 UTC m=+25.752782782"
Jul 18 00:40:56.169700 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:56.169663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:56.169992 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:56.169663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:56.169992 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:56.169787    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:56.169992 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:56.169856    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:56.169992 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:56.169670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:56.169992 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:56.169964    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:56.245235 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:56.245199    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:56.245863 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:56.245842    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:56.420777 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:56.420694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"e6c4d1f8d6b04d7dd15139556f240512a854e17b4f485ddb217a955b94dce9ee"}
Jul 18 00:40:56.421498 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:56.421448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ghhg9"
Jul 18 00:40:58.170238 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.170015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:58.170988 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.170015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:58.170988 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:58.170320    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:40:58.170988 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:58.170395    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:58.170988 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.170050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:58.170988 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:58.170489    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:58.428951 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.428864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" event={"ID":"ebf3259a-976a-4141-a2d9-456552d62d1a","Type":"ContainerStarted","Data":"d471114dec489289d95102a5b87d1e1ca7cde2840c5b842c4d06e2fdb4a8f324"}
Jul 18 00:40:58.429150 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.429103    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:58.429231 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.429172    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:58.430710 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.430678    2563 generic.go:356] "Generic (PLEG): container finished" podID="02374191-e591-4213-b783-3fc5d47e9625" containerID="2f294f966c025fc5e7d95e3b502c9c9edc1288dcd8738b9966823b04716763ca" exitCode=0
Jul 18 00:40:58.430821 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.430754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" event={"ID":"02374191-e591-4213-b783-3fc5d47e9625","Type":"ContainerDied","Data":"2f294f966c025fc5e7d95e3b502c9c9edc1288dcd8738b9966823b04716763ca"}
Jul 18 00:40:58.444882 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.444860    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:58.456405 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:58.456359    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6" podStartSLOduration=8.077635125 podStartE2EDuration="27.456343625s" totalImagesPullingTime="19.3787085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:58.454119799 +0000 UTC m=+28.774097184" watchObservedRunningTime="2026-07-18 00:40:58.456343625 +0000 UTC m=+28.776321008"
Jul 18 00:40:59.433302 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:59.433068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:59.447498 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:59.447465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:40:59.735275 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:59.735248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9b8ml"]
Jul 18 00:40:59.735498 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:59.735367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:40:59.735498 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:59.735469    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:40:59.737911 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:59.737885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vv5qj"]
Jul 18 00:40:59.738077 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:59.737982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:40:59.738150 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:59.738086    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:40:59.738432 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:59.738409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g5fq8"]
Jul 18 00:40:59.738543 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:40:59.738503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:40:59.738601 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:40:59.738586    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:41:00.436513 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:00.436462    2563 generic.go:356] "Generic (PLEG): container finished" podID="02374191-e591-4213-b783-3fc5d47e9625" containerID="3e1deb862209cfeb2cd759cd99b8b8ab26dec569adf79f6ef55e2e1848e7b9d2" exitCode=0
Jul 18 00:41:00.437012 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:00.436507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" event={"ID":"02374191-e591-4213-b783-3fc5d47e9625","Type":"ContainerDied","Data":"3e1deb862209cfeb2cd759cd99b8b8ab26dec569adf79f6ef55e2e1848e7b9d2"}
Jul 18 00:41:01.169439 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:01.169405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:01.169639 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:01.169448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:41:01.169639 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:01.169447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:41:01.169639 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:01.169531    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:41:01.169639 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:01.169631    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:41:01.169842 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:01.169728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:41:02.443644 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:02.443604    2563 generic.go:356] "Generic (PLEG): container finished" podID="02374191-e591-4213-b783-3fc5d47e9625" containerID="44900f53f22a64a63ec7fdf01ae6094012f9b7ce23cccbcc07a2d445d92cf0bb" exitCode=0
Jul 18 00:41:02.444128 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:02.443663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" event={"ID":"02374191-e591-4213-b783-3fc5d47e9625","Type":"ContainerDied","Data":"44900f53f22a64a63ec7fdf01ae6094012f9b7ce23cccbcc07a2d445d92cf0bb"}
Jul 18 00:41:03.170035 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:03.170004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:41:03.170243 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:03.170002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:41:03.170243 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:03.170158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:03.170243 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:03.170151    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:41:03.170421 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:03.170277    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:41:03.170421 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:03.170338    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:41:04.022944 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:04.022911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:41:04.023420 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:04.022969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:41:04.023420 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:04.023094    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 00:41:04.023420 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:04.023100    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:41:04.023420 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:04.023154    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret podName:2a5b6fba-2c56-4526-9e0f-21ca35cba6f2 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:36.023137759 +0000 UTC m=+66.343115121 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret") pod "global-pull-secret-syncer-g5fq8" (UID: "2a5b6fba-2c56-4526-9e0f-21ca35cba6f2") : object "kube-system"/"original-pull-secret" not registered
Jul 18 00:41:04.023420 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:04.023171    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs podName:75ee96b7-bcc7-4f75-bf11-52a674359b51 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:36.023164693 +0000 UTC m=+66.343142054 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs") pod "network-metrics-daemon-9b8ml" (UID: "75ee96b7-bcc7-4f75-bf11-52a674359b51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 00:41:04.124193 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:04.124149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:04.124368 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:04.124331    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 00:41:04.124368 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:04.124355    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 00:41:04.124368 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:04.124364    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tt5cb for pod openshift-network-diagnostics/network-check-target-vv5qj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:41:04.124521 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:04.124423    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb podName:509a8c3e-3258-4abb-83a6-ad5bb4722523 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:36.124405548 +0000 UTC m=+66.444382919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tt5cb" (UniqueName: "kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb") pod "network-check-target-vv5qj" (UID: "509a8c3e-3258-4abb-83a6-ad5bb4722523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 00:41:05.170013 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.169809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:41:05.170450 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.169809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:05.170450 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:05.170128    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5fq8" podUID="2a5b6fba-2c56-4526-9e0f-21ca35cba6f2"
Jul 18 00:41:05.170450 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.169809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:41:05.170450 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:05.170162    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vv5qj" podUID="509a8c3e-3258-4abb-83a6-ad5bb4722523"
Jul 18 00:41:05.170450 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:05.170286    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9b8ml" podUID="75ee96b7-bcc7-4f75-bf11-52a674359b51"
Jul 18 00:41:05.508840 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.508810    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-142.ec2.internal" event="NodeReady"
Jul 18 00:41:05.509034 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.508964    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 18 00:41:05.552237 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.552204    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f7f84d98c-hf2wv"]
Jul 18 00:41:05.554686 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.554663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.557470 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.557446    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hd2wz\""
Jul 18 00:41:05.557470 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.557473    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 18 00:41:05.557678 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.557492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 18 00:41:05.557678 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.557588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 18 00:41:05.562849 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.562819    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r7sw4"]
Jul 18 00:41:05.564199 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.564180    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 18 00:41:05.564976 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.564959    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jzfr6"]
Jul 18 00:41:05.565127 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.565108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r7sw4"
Jul 18 00:41:05.567146 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.567123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f7f84d98c-hf2wv"]
Jul 18 00:41:05.567275 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.567215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.568256 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.568217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 18 00:41:05.568833 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.568471    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h5lw6\""
Jul 18 00:41:05.568833 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.568536    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 18 00:41:05.568833 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.568687    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 18 00:41:05.569649 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.569631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 18 00:41:05.569916 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.569887    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 18 00:41:05.569916 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.569915    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7vqp\""
Jul 18 00:41:05.581179 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.581103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r7sw4"]
Jul 18 00:41:05.582020 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.581998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jzfr6"]
Jul 18 00:41:05.736917 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.736880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-installation-pull-secrets\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.737133 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.736926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9qf\" (UniqueName: \"kubernetes.io/projected/112acebf-c853-4157-9d00-93c27387440f-kube-api-access-mv9qf\") pod \"ingress-canary-r7sw4\" (UID: \"112acebf-c853-4157-9d00-93c27387440f\") " pod="openshift-ingress-canary/ingress-canary-r7sw4"
Jul 18 00:41:05.737133 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.736949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea76c39f-22ad-404a-8a10-c0ec2df2697d-ca-trust-extracted\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.737133 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112acebf-c853-4157-9d00-93c27387440f-cert\") pod \"ingress-canary-r7sw4\" (UID: \"112acebf-c853-4157-9d00-93c27387440f\") " pod="openshift-ingress-canary/ingress-canary-r7sw4"
Jul 18 00:41:05.737133 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-bound-sa-token\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.737133 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab019bac-1682-4e2c-9167-894dc2ad5cb1-config-volume\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.737403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.737403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-trusted-ca\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.737403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab019bac-1682-4e2c-9167-894dc2ad5cb1-tmp-dir\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.737403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.737403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4cr\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-kube-api-access-kh4cr\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.737403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-certificates\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.737403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckh2\" (UniqueName: \"kubernetes.io/projected/ab019bac-1682-4e2c-9167-894dc2ad5cb1-kube-api-access-2ckh2\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.737403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.737357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-image-registry-private-configuration\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838328 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-trusted-ca\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838328 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab019bac-1682-4e2c-9167-894dc2ad5cb1-tmp-dir\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.838328 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4cr\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-kube-api-access-kh4cr\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-certificates\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckh2\" (UniqueName: \"kubernetes.io/projected/ab019bac-1682-4e2c-9167-894dc2ad5cb1-kube-api-access-2ckh2\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-image-registry-private-configuration\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-installation-pull-secrets\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9qf\" (UniqueName: \"kubernetes.io/projected/112acebf-c853-4157-9d00-93c27387440f-kube-api-access-mv9qf\") pod \"ingress-canary-r7sw4\" (UID: \"112acebf-c853-4157-9d00-93c27387440f\") " pod="openshift-ingress-canary/ingress-canary-r7sw4"
Jul 18 00:41:05.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea76c39f-22ad-404a-8a10-c0ec2df2697d-ca-trust-extracted\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112acebf-c853-4157-9d00-93c27387440f-cert\") pod \"ingress-canary-r7sw4\" (UID: \"112acebf-c853-4157-9d00-93c27387440f\") " pod="openshift-ingress-canary/ingress-canary-r7sw4"
Jul 18 00:41:05.838972 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-bound-sa-token\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.838972 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab019bac-1682-4e2c-9167-894dc2ad5cb1-config-volume\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.838972 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.838972 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.838793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab019bac-1682-4e2c-9167-894dc2ad5cb1-tmp-dir\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.838972 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:05.838899    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 00:41:05.838972 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:05.838967    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls podName:ab019bac-1682-4e2c-9167-894dc2ad5cb1 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:06.338947731 +0000 UTC m=+36.658925102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls") pod "dns-default-jzfr6" (UID: "ab019bac-1682-4e2c-9167-894dc2ad5cb1") : secret "dns-default-metrics-tls" not found
Jul 18 00:41:05.839440 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:05.839419    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 00:41:05.839509 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:05.839444    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7f84d98c-hf2wv: secret "image-registry-tls" not found
Jul 18 00:41:05.839509 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:05.839496    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls podName:ea76c39f-22ad-404a-8a10-c0ec2df2697d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:06.339477844 +0000 UTC m=+36.659455214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls") pod "image-registry-5f7f84d98c-hf2wv" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d") : secret "image-registry-tls" not found
Jul 18 00:41:05.839611 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.839593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab019bac-1682-4e2c-9167-894dc2ad5cb1-config-volume\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.839663 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.839617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-certificates\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.839755 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.839734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-trusted-ca\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.839861 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.839838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea76c39f-22ad-404a-8a10-c0ec2df2697d-ca-trust-extracted\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.843799 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.843776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-installation-pull-secrets\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.843946 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.843778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-image-registry-private-configuration\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.843946 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.843849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112acebf-c853-4157-9d00-93c27387440f-cert\") pod \"ingress-canary-r7sw4\" (UID: \"112acebf-c853-4157-9d00-93c27387440f\") " pod="openshift-ingress-canary/ingress-canary-r7sw4"
Jul 18 00:41:05.847970 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.847945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-bound-sa-token\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.848152 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.848116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9qf\" (UniqueName: \"kubernetes.io/projected/112acebf-c853-4157-9d00-93c27387440f-kube-api-access-mv9qf\") pod \"ingress-canary-r7sw4\" (UID: \"112acebf-c853-4157-9d00-93c27387440f\") " pod="openshift-ingress-canary/ingress-canary-r7sw4"
Jul 18 00:41:05.848550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.848520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckh2\" (UniqueName: \"kubernetes.io/projected/ab019bac-1682-4e2c-9167-894dc2ad5cb1-kube-api-access-2ckh2\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:05.848628 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.848530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4cr\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-kube-api-access-kh4cr\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:05.876336 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:05.876299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r7sw4"
Jul 18 00:41:06.342968 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:06.342904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:06.343600 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:06.343032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:06.343600 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:06.343099    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 00:41:06.343600 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:06.343118    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7f84d98c-hf2wv: secret "image-registry-tls" not found
Jul 18 00:41:06.343600 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:06.343182    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls podName:ea76c39f-22ad-404a-8a10-c0ec2df2697d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:07.343164629 +0000 UTC m=+37.663142012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls") pod "image-registry-5f7f84d98c-hf2wv" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d") : secret "image-registry-tls" not found
Jul 18 00:41:06.343600 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:06.343207    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 00:41:06.343600 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:06.343289    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls podName:ab019bac-1682-4e2c-9167-894dc2ad5cb1 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:07.343269906 +0000 UTC m=+37.663247267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls") pod "dns-default-jzfr6" (UID: "ab019bac-1682-4e2c-9167-894dc2ad5cb1") : secret "dns-default-metrics-tls" not found
Jul 18 00:41:06.506165 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:06.505966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r7sw4"]
Jul 18 00:41:06.510210 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:06.510181    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112acebf_c853_4157_9d00_93c27387440f.slice/crio-f74c8dac517d648a22bda0277f9db17274730b39de346c9c99a814b93bd62000 WatchSource:0}: Error finding container f74c8dac517d648a22bda0277f9db17274730b39de346c9c99a814b93bd62000: Status 404 returned error can't find the container with id f74c8dac517d648a22bda0277f9db17274730b39de346c9c99a814b93bd62000
Jul 18 00:41:07.169777 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.169740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:41:07.169955 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.169860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:07.169955 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.169899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:41:07.172517 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.172482    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 18 00:41:07.173492 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.173467    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9tdp4\""
Jul 18 00:41:07.173610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.173504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 18 00:41:07.173610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.173516    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 18 00:41:07.173610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.173476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2stqj\""
Jul 18 00:41:07.173754 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.173467    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 18 00:41:07.354199 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.354150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:07.354941 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.354250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:07.354941 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:07.354381    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 00:41:07.354941 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:07.354419    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 00:41:07.354941 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:07.354434    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7f84d98c-hf2wv: secret "image-registry-tls" not found
Jul 18 00:41:07.354941 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:07.354452    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls podName:ab019bac-1682-4e2c-9167-894dc2ad5cb1 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:09.35443575 +0000 UTC m=+39.674413124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls") pod "dns-default-jzfr6" (UID: "ab019bac-1682-4e2c-9167-894dc2ad5cb1") : secret "dns-default-metrics-tls" not found
Jul 18 00:41:07.354941 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:07.354493    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls podName:ea76c39f-22ad-404a-8a10-c0ec2df2697d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:09.35447691 +0000 UTC m=+39.674454277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls") pod "image-registry-5f7f84d98c-hf2wv" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d") : secret "image-registry-tls" not found
Jul 18 00:41:07.459793 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.459759    2563 generic.go:356] "Generic (PLEG): container finished" podID="02374191-e591-4213-b783-3fc5d47e9625" containerID="e6b174f938c7cbfd1cf86365929cbd195ba227383a0aaf6149b2186d551ad1fc" exitCode=0
Jul 18 00:41:07.459951 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.459840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" event={"ID":"02374191-e591-4213-b783-3fc5d47e9625","Type":"ContainerDied","Data":"e6b174f938c7cbfd1cf86365929cbd195ba227383a0aaf6149b2186d551ad1fc"}
Jul 18 00:41:07.461553 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:07.461529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r7sw4" event={"ID":"112acebf-c853-4157-9d00-93c27387440f","Type":"ContainerStarted","Data":"f74c8dac517d648a22bda0277f9db17274730b39de346c9c99a814b93bd62000"}
Jul 18 00:41:08.466741 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:08.466709    2563 generic.go:356] "Generic (PLEG): container finished" podID="02374191-e591-4213-b783-3fc5d47e9625" containerID="b4c3dffb65ccf2d68e3da5a40a1ddd3f78c82370fb1786d99818a454864c2934" exitCode=0
Jul 18 00:41:08.467135 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:08.466756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" event={"ID":"02374191-e591-4213-b783-3fc5d47e9625","Type":"ContainerDied","Data":"b4c3dffb65ccf2d68e3da5a40a1ddd3f78c82370fb1786d99818a454864c2934"}
Jul 18 00:41:09.369799 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:09.369757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:09.369956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:09.369843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:09.369956 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:09.369924    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 00:41:09.369956 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:09.369948    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7f84d98c-hf2wv: secret "image-registry-tls" not found
Jul 18 00:41:09.370047 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:09.369960    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 00:41:09.370047 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:09.370006    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls podName:ab019bac-1682-4e2c-9167-894dc2ad5cb1 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:13.369991232 +0000 UTC m=+43.689968592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls") pod "dns-default-jzfr6" (UID: "ab019bac-1682-4e2c-9167-894dc2ad5cb1") : secret "dns-default-metrics-tls" not found
Jul 18 00:41:09.370047 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:09.370019    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls podName:ea76c39f-22ad-404a-8a10-c0ec2df2697d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:13.370013148 +0000 UTC m=+43.689990509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls") pod "image-registry-5f7f84d98c-hf2wv" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d") : secret "image-registry-tls" not found
Jul 18 00:41:09.470478 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:09.470443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r7sw4" event={"ID":"112acebf-c853-4157-9d00-93c27387440f","Type":"ContainerStarted","Data":"86efacfc412923b21a0e43276f4d2d5ee48d7271612578c5794fc7bdedffa342"}
Jul 18 00:41:09.473735 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:09.473709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" event={"ID":"02374191-e591-4213-b783-3fc5d47e9625","Type":"ContainerStarted","Data":"ea19363fc1fac769cebce5ed05c47dbcd95c947c72c2135415915c2497f723c1"}
Jul 18 00:41:09.486224 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:09.486173    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r7sw4" podStartSLOduration=2.5053235320000002 podStartE2EDuration="4.486155014s" totalImagesPullingTime="1.980831482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:09.485812694 +0000 UTC m=+39.805790127" watchObservedRunningTime="2026-07-18 00:41:09.486155014 +0000 UTC m=+39.806132399"
Jul 18 00:41:09.510237 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:09.510186    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k4lcb" podStartSLOduration=8.796642231 podStartE2EDuration="38.510174319s" totalImagesPullingTime="29.713532088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:09.509110799 +0000 UTC m=+39.829088178" watchObservedRunningTime="2026-07-18 00:41:09.510174319 +0000 UTC m=+39.830151701"
Jul 18 00:41:13.395775 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:13.395728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:13.396215 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:13.395789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:13.396215 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:13.395884    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 00:41:13.396215 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:13.395955    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls podName:ab019bac-1682-4e2c-9167-894dc2ad5cb1 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:21.395937719 +0000 UTC m=+51.715915080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls") pod "dns-default-jzfr6" (UID: "ab019bac-1682-4e2c-9167-894dc2ad5cb1") : secret "dns-default-metrics-tls" not found
Jul 18 00:41:13.396215 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:13.395887    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 00:41:13.396215 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:13.395996    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7f84d98c-hf2wv: secret "image-registry-tls" not found
Jul 18 00:41:13.396215 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:13.396040    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls podName:ea76c39f-22ad-404a-8a10-c0ec2df2697d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:21.396028011 +0000 UTC m=+51.716005372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls") pod "image-registry-5f7f84d98c-hf2wv" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d") : secret "image-registry-tls" not found
Jul 18 00:41:15.576231 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.576197    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n"]
Jul 18 00:41:15.602624 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.602584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n"]
Jul 18 00:41:15.602777 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.602693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n"
Jul 18 00:41:15.604936 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.604912    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 18 00:41:15.606023 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.606001    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6zl45\""
Jul 18 00:41:15.606130 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.606013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 18 00:41:15.609488 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.609465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6s7\" (UniqueName: \"kubernetes.io/projected/6123a428-e436-4a58-a29d-e3f9c671bbc1-kube-api-access-nb6s7\") pod \"migrator-7cc9c7ffd7-hlx4n\" (UID: \"6123a428-e436-4a58-a29d-e3f9c671bbc1\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n"
Jul 18 00:41:15.709989 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.709949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6s7\" (UniqueName: \"kubernetes.io/projected/6123a428-e436-4a58-a29d-e3f9c671bbc1-kube-api-access-nb6s7\") pod \"migrator-7cc9c7ffd7-hlx4n\" (UID: \"6123a428-e436-4a58-a29d-e3f9c671bbc1\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n"
Jul 18 00:41:15.719030 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.719000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6s7\" (UniqueName: \"kubernetes.io/projected/6123a428-e436-4a58-a29d-e3f9c671bbc1-kube-api-access-nb6s7\") pod \"migrator-7cc9c7ffd7-hlx4n\" (UID: \"6123a428-e436-4a58-a29d-e3f9c671bbc1\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n"
Jul 18 00:41:15.911577 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.911486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n"
Jul 18 00:41:15.964113 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.964075    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4zgt4"]
Jul 18 00:41:15.986905 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.986787    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4zgt4"]
Jul 18 00:41:15.987223 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.987202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:15.989718 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.989660    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 18 00:41:15.990175 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.990041    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 18 00:41:15.990338 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.990317    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 18 00:41:15.990730 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.990685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cqrgt\""
Jul 18 00:41:15.990730 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:15.990694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 18 00:41:16.012438 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.012413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.012571 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.012455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82f6fc6a-2500-4c10-860c-2b2699f81669-data-volume\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.012571 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.012480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.012654 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.012590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hth6\" (UniqueName: \"kubernetes.io/projected/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-api-access-8hth6\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.012654 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.012641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82f6fc6a-2500-4c10-860c-2b2699f81669-crio-socket\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.050199 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.048806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n"]
Jul 18 00:41:16.113541 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.113500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.113541 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.113545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82f6fc6a-2500-4c10-860c-2b2699f81669-data-volume\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.113836 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.113565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.113836 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:16.113624    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 00:41:16.113836 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:16.113673    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:16.113836 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:16.113683    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:16.613667139 +0000 UTC m=+46.933644503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : configmap "kube-rbac-proxy" not found
Jul 18 00:41:16.113836 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.113790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hth6\" (UniqueName: \"kubernetes.io/projected/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-api-access-8hth6\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.113836 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:16.113835    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:16.613814647 +0000 UTC m=+46.933792011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:16.114216 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.113893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82f6fc6a-2500-4c10-860c-2b2699f81669-crio-socket\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.114216 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.114090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82f6fc6a-2500-4c10-860c-2b2699f81669-crio-socket\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.114296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.114220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82f6fc6a-2500-4c10-860c-2b2699f81669-data-volume\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.122329 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.122287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hth6\" (UniqueName: \"kubernetes.io/projected/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-api-access-8hth6\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.490515 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.490473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n" event={"ID":"6123a428-e436-4a58-a29d-e3f9c671bbc1","Type":"ContainerStarted","Data":"2263f12e9f1288e702ae28a302766641b8838ab22d993bb03b7f3c422573d4ae"}
Jul 18 00:41:16.618705 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.618661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.619199 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:16.618726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:16.619199 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:16.618829    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:16.619199 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:16.618872    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 00:41:16.619199 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:16.618926    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:17.618905304 +0000 UTC m=+47.938882682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:16.619199 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:16.618946    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:17.61893644 +0000 UTC m=+47.938913805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : configmap "kube-rbac-proxy" not found
Jul 18 00:41:17.158987 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.158948    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-sn9xq"]
Jul 18 00:41:17.185372 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.185343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-sn9xq"]
Jul 18 00:41:17.185540 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.185454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.188757 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.188730    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 18 00:41:17.188757 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.188749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 18 00:41:17.188952 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.188730    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 18 00:41:17.189686 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.189651    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6bp99\""
Jul 18 00:41:17.189686 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.189678    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 18 00:41:17.223402 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.223369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vvw\" (UniqueName: \"kubernetes.io/projected/9b569f8d-8300-40ae-b4e5-de0136140436-kube-api-access-q8vvw\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.223573 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.223455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b569f8d-8300-40ae-b4e5-de0136140436-signing-cabundle\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.223573 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.223510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b569f8d-8300-40ae-b4e5-de0136140436-signing-key\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.324702 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.324663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vvw\" (UniqueName: \"kubernetes.io/projected/9b569f8d-8300-40ae-b4e5-de0136140436-kube-api-access-q8vvw\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.324866 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.324754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b569f8d-8300-40ae-b4e5-de0136140436-signing-cabundle\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.324866 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.324809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b569f8d-8300-40ae-b4e5-de0136140436-signing-key\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.325464 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.325445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b569f8d-8300-40ae-b4e5-de0136140436-signing-cabundle\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.327087 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.327050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b569f8d-8300-40ae-b4e5-de0136140436-signing-key\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.332606 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.332577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vvw\" (UniqueName: \"kubernetes.io/projected/9b569f8d-8300-40ae-b4e5-de0136140436-kube-api-access-q8vvw\") pod \"service-ca-58598695f7-sn9xq\" (UID: \"9b569f8d-8300-40ae-b4e5-de0136140436\") " pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.494578 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.494543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-sn9xq"
Jul 18 00:41:17.627679 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.627648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:17.628035 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.627707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:17.628035 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:17.627816    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:17.628035 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:17.627815    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 00:41:17.628035 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:17.627881    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:19.627860098 +0000 UTC m=+49.947837470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:17.628035 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:17.627934    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:19.627908703 +0000 UTC m=+49.947886076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : configmap "kube-rbac-proxy" not found
Jul 18 00:41:17.672917 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:17.672890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-sn9xq"]
Jul 18 00:41:17.676112 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:17.676088    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b569f8d_8300_40ae_b4e5_de0136140436.slice/crio-cf61f3a99d57b8304c949ef5f54a0f1e2e65850d5f27052ad649032c7ffbd4b8 WatchSource:0}: Error finding container cf61f3a99d57b8304c949ef5f54a0f1e2e65850d5f27052ad649032c7ffbd4b8: Status 404 returned error can't find the container with id cf61f3a99d57b8304c949ef5f54a0f1e2e65850d5f27052ad649032c7ffbd4b8
Jul 18 00:41:18.498246 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:18.498141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n" event={"ID":"6123a428-e436-4a58-a29d-e3f9c671bbc1","Type":"ContainerStarted","Data":"28d1fa903f6c332ac8aea8d84b791c877df1018b352711447750c7865c73523a"}
Jul 18 00:41:18.498246 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:18.498208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n" event={"ID":"6123a428-e436-4a58-a29d-e3f9c671bbc1","Type":"ContainerStarted","Data":"7da9bd858da1eeb87809bc573504b875b95612766f6a965310fe1b8d8e81a404"}
Jul 18 00:41:18.499645 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:18.499611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-sn9xq" event={"ID":"9b569f8d-8300-40ae-b4e5-de0136140436","Type":"ContainerStarted","Data":"cf61f3a99d57b8304c949ef5f54a0f1e2e65850d5f27052ad649032c7ffbd4b8"}
Jul 18 00:41:18.519816 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:18.519758    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hlx4n" podStartSLOduration=2.02020376 podStartE2EDuration="3.519743128s" totalImagesPullingTime="1.499539368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:18.519236862 +0000 UTC m=+48.839214247" watchObservedRunningTime="2026-07-18 00:41:18.519743128 +0000 UTC m=+48.839720562"
Jul 18 00:41:18.554227 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:18.554197    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgnlq_6c200b84-577b-45d8-a6b6-e715ce2d4262/dns-node-resolver/0.log"
Jul 18 00:41:19.557439 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:19.557410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tkd94_beeaa768-6025-4191-9306-1c75abdd1477/node-ca/0.log"
Jul 18 00:41:19.645080 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:19.645013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:19.645270 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:19.645137    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 00:41:19.645270 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:19.645185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:19.645270 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:19.645206    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:23.64518563 +0000 UTC m=+53.965163009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : configmap "kube-rbac-proxy" not found
Jul 18 00:41:19.645425 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:19.645323    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:19.645425 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:19.645370    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:23.645357148 +0000 UTC m=+53.965334513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:20.154154 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:20.154065    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7sw4_112acebf-c853-4157-9d00-93c27387440f/serve-healthcheck-canary/0.log"
Jul 18 00:41:20.354039 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:20.354013    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-hlx4n_6123a428-e436-4a58-a29d-e3f9c671bbc1/migrator/0.log"
Jul 18 00:41:20.506525 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:20.506478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-sn9xq" event={"ID":"9b569f8d-8300-40ae-b4e5-de0136140436","Type":"ContainerStarted","Data":"b2c5b031cfccf35a05253b17d8493cf6ddd19d8982b60caabe990269e917803f"}
Jul 18 00:41:20.530570 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:20.530510    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-sn9xq" podStartSLOduration=1.388283694 podStartE2EDuration="3.530488764s" totalImagesPullingTime="2.14220507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:20.529620456 +0000 UTC m=+50.849597840" watchObservedRunningTime="2026-07-18 00:41:20.530488764 +0000 UTC m=+50.850466148"
Jul 18 00:41:20.554787 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:20.554756    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-hlx4n_6123a428-e436-4a58-a29d-e3f9c671bbc1/graceful-termination/0.log"
Jul 18 00:41:21.458852 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:21.458804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:21.459322 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:21.458937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:21.459322 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:21.458977    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 00:41:21.459322 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:21.459003    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7f84d98c-hf2wv: secret "image-registry-tls" not found
Jul 18 00:41:21.459322 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:21.459067    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 00:41:21.459322 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:21.459086    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls podName:ea76c39f-22ad-404a-8a10-c0ec2df2697d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:37.459049399 +0000 UTC m=+67.779026760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls") pod "image-registry-5f7f84d98c-hf2wv" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d") : secret "image-registry-tls" not found
Jul 18 00:41:21.459322 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:21.459118    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls podName:ab019bac-1682-4e2c-9167-894dc2ad5cb1 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:37.459101482 +0000 UTC m=+67.779078849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls") pod "dns-default-jzfr6" (UID: "ab019bac-1682-4e2c-9167-894dc2ad5cb1") : secret "dns-default-metrics-tls" not found
Jul 18 00:41:23.676507 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:23.676464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:23.676507 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:23.676512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:23.677023 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:23.676628    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:23.677023 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:23.676717    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls podName:82f6fc6a-2500-4c10-860c-2b2699f81669 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:31.676682094 +0000 UTC m=+61.996659458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4zgt4" (UID: "82f6fc6a-2500-4c10-860c-2b2699f81669") : secret "insights-runtime-extractor-tls" not found
Jul 18 00:41:23.677149 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:23.677030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82f6fc6a-2500-4c10-860c-2b2699f81669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:31.449468 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:31.449438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clxs6"
Jul 18 00:41:31.741020 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:31.740933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:31.743356 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:31.743319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82f6fc6a-2500-4c10-860c-2b2699f81669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4zgt4\" (UID: \"82f6fc6a-2500-4c10-860c-2b2699f81669\") " pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:31.900228 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:31.900189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cqrgt\""
Jul 18 00:41:31.908134 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:31.908105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4zgt4"
Jul 18 00:41:32.032208 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:32.032123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4zgt4"]
Jul 18 00:41:32.035488 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:32.035460    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f6fc6a_2500_4c10_860c_2b2699f81669.slice/crio-ce8647a4a9985d1cb9f1000d5877d1157a8c673668d051da49e5805c213fea07 WatchSource:0}: Error finding container ce8647a4a9985d1cb9f1000d5877d1157a8c673668d051da49e5805c213fea07: Status 404 returned error can't find the container with id ce8647a4a9985d1cb9f1000d5877d1157a8c673668d051da49e5805c213fea07
Jul 18 00:41:32.538505 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:32.538452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4zgt4" event={"ID":"82f6fc6a-2500-4c10-860c-2b2699f81669","Type":"ContainerStarted","Data":"ce8647a4a9985d1cb9f1000d5877d1157a8c673668d051da49e5805c213fea07"}
Jul 18 00:41:36.076257 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.076213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:41:36.076721 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.076359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:41:36.080622 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.080593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 18 00:41:36.080622 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.080611    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 18 00:41:36.088697 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.088670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a5b6fba-2c56-4526-9e0f-21ca35cba6f2-original-pull-secret\") pod \"global-pull-secret-syncer-g5fq8\" (UID: \"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2\") " pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:41:36.088926 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.088731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75ee96b7-bcc7-4f75-bf11-52a674359b51-metrics-certs\") pod \"network-metrics-daemon-9b8ml\" (UID: \"75ee96b7-bcc7-4f75-bf11-52a674359b51\") " pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:41:36.177544 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.177511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:36.180141 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.180120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 18 00:41:36.190309 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.190289    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 18 00:41:36.201684 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.201654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5cb\" (UniqueName: \"kubernetes.io/projected/509a8c3e-3258-4abb-83a6-ad5bb4722523-kube-api-access-tt5cb\") pod \"network-check-target-vv5qj\" (UID: \"509a8c3e-3258-4abb-83a6-ad5bb4722523\") " pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:36.281152 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.281110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5fq8"
Jul 18 00:41:36.291246 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.291123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2stqj\""
Jul 18 00:41:36.296164 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.296134    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9tdp4\""
Jul 18 00:41:36.299003 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.298979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:36.305128 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:36.305105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9b8ml"
Jul 18 00:41:37.081516 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.081387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vv5qj"]
Jul 18 00:41:37.085040 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:37.085011    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509a8c3e_3258_4abb_83a6_ad5bb4722523.slice/crio-27d4991fbf886e065818af4eb47efa6ec0ef994d379854258b773afb1426636c WatchSource:0}: Error finding container 27d4991fbf886e065818af4eb47efa6ec0ef994d379854258b773afb1426636c: Status 404 returned error can't find the container with id 27d4991fbf886e065818af4eb47efa6ec0ef994d379854258b773afb1426636c
Jul 18 00:41:37.310551 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.310272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g5fq8"]
Jul 18 00:41:37.311239 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.311218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9b8ml"]
Jul 18 00:41:37.314154 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:37.314118    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ee96b7_bcc7_4f75_bf11_52a674359b51.slice/crio-e35fde7a1b8e9cd1e43ae46b9cc5cc73f2b38254b4611cede35f880151fd9537 WatchSource:0}: Error finding container e35fde7a1b8e9cd1e43ae46b9cc5cc73f2b38254b4611cede35f880151fd9537: Status 404 returned error can't find the container with id e35fde7a1b8e9cd1e43ae46b9cc5cc73f2b38254b4611cede35f880151fd9537
Jul 18 00:41:37.314686 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:37.314662    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5b6fba_2c56_4526_9e0f_21ca35cba6f2.slice/crio-30652a9be872885322f7d03e978bb699d671e46cebcc05c89f0c1ce077981cb1 WatchSource:0}: Error finding container 30652a9be872885322f7d03e978bb699d671e46cebcc05c89f0c1ce077981cb1: Status 404 returned error can't find the container with id 30652a9be872885322f7d03e978bb699d671e46cebcc05c89f0c1ce077981cb1
Jul 18 00:41:37.486993 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.486956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:37.487162 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.487017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:37.489399 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.489380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab019bac-1682-4e2c-9167-894dc2ad5cb1-metrics-tls\") pod \"dns-default-jzfr6\" (UID: \"ab019bac-1682-4e2c-9167-894dc2ad5cb1\") " pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:37.489467 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.489419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"image-registry-5f7f84d98c-hf2wv\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") " pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:37.555800 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.555764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4zgt4" event={"ID":"82f6fc6a-2500-4c10-860c-2b2699f81669","Type":"ContainerStarted","Data":"957414476ac5c70bc47a48d52d65306885ad38f9e893c1dca4190971d2de1ead"}
Jul 18 00:41:37.557222 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.557186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9b8ml" event={"ID":"75ee96b7-bcc7-4f75-bf11-52a674359b51","Type":"ContainerStarted","Data":"e35fde7a1b8e9cd1e43ae46b9cc5cc73f2b38254b4611cede35f880151fd9537"}
Jul 18 00:41:37.558436 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.558410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vv5qj" event={"ID":"509a8c3e-3258-4abb-83a6-ad5bb4722523","Type":"ContainerStarted","Data":"27d4991fbf886e065818af4eb47efa6ec0ef994d379854258b773afb1426636c"}
Jul 18 00:41:37.559538 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.559518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g5fq8" event={"ID":"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2","Type":"ContainerStarted","Data":"30652a9be872885322f7d03e978bb699d671e46cebcc05c89f0c1ce077981cb1"}
Jul 18 00:41:37.668127 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.668034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hd2wz\""
Jul 18 00:41:37.677067 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.677018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:37.685345 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.685313    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7vqp\""
Jul 18 00:41:37.693784 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.693761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:37.855781 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.855741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f7f84d98c-hf2wv"]
Jul 18 00:41:37.861024 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:37.860989    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea76c39f_22ad_404a_8a10_c0ec2df2697d.slice/crio-0f450df9105bc0f6f0452704fbc4bcc6fcfc1c385cad58f7671f8d062a799552 WatchSource:0}: Error finding container 0f450df9105bc0f6f0452704fbc4bcc6fcfc1c385cad58f7671f8d062a799552: Status 404 returned error can't find the container with id 0f450df9105bc0f6f0452704fbc4bcc6fcfc1c385cad58f7671f8d062a799552
Jul 18 00:41:37.883149 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:37.883115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jzfr6"]
Jul 18 00:41:37.887953 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:37.887923    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab019bac_1682_4e2c_9167_894dc2ad5cb1.slice/crio-62b2f8db9b7392b73d51cee55f573c0fa4d6656287c7a5743c6407c1e6af0802 WatchSource:0}: Error finding container 62b2f8db9b7392b73d51cee55f573c0fa4d6656287c7a5743c6407c1e6af0802: Status 404 returned error can't find the container with id 62b2f8db9b7392b73d51cee55f573c0fa4d6656287c7a5743c6407c1e6af0802
Jul 18 00:41:38.576739 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:38.576636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzfr6" event={"ID":"ab019bac-1682-4e2c-9167-894dc2ad5cb1","Type":"ContainerStarted","Data":"62b2f8db9b7392b73d51cee55f573c0fa4d6656287c7a5743c6407c1e6af0802"}
Jul 18 00:41:38.580657 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:38.580009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv" event={"ID":"ea76c39f-22ad-404a-8a10-c0ec2df2697d","Type":"ContainerStarted","Data":"a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c"}
Jul 18 00:41:38.580657 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:38.580085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv" event={"ID":"ea76c39f-22ad-404a-8a10-c0ec2df2697d","Type":"ContainerStarted","Data":"0f450df9105bc0f6f0452704fbc4bcc6fcfc1c385cad58f7671f8d062a799552"}
Jul 18 00:41:38.580657 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:38.580374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:41:38.602284 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:38.602190    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv" podStartSLOduration=58.602169396 podStartE2EDuration="58.602169396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:38.601142926 +0000 UTC m=+68.921120309" watchObservedRunningTime="2026-07-18 00:41:38.602169396 +0000 UTC m=+68.922146786"
Jul 18 00:41:40.498794 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.498757    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"]
Jul 18 00:41:40.505025 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.504991    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"]
Jul 18 00:41:40.505197 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.505133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.508775 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.508678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"
Jul 18 00:41:40.508922 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.508877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 18 00:41:40.510202 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.509911    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 18 00:41:40.510202 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.509938    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 18 00:41:40.510202 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.509938    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 18 00:41:40.512228 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.512208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8c4vw\""
Jul 18 00:41:40.513092 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.513072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 18 00:41:40.518974 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.518468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"]
Jul 18 00:41:40.519933 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.519907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"]
Jul 18 00:41:40.599086 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.598969    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-9gd7s"]
Jul 18 00:41:40.606239 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.606201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-9gd7s"
Jul 18 00:41:40.609087 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.608841    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 18 00:41:40.609087 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.608930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-h6x2l\""
Jul 18 00:41:40.609378 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.609146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 18 00:41:40.615993 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.615562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9gk\" (UniqueName: \"kubernetes.io/projected/7b0afb87-2c01-47c3-aa0d-bc489c69a655-kube-api-access-tz9gk\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.615993 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.615635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd7f0137-a7f4-44ee-8642-2c95763c557b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74cbcb796b-bm8nl\" (UID: \"cd7f0137-a7f4-44ee-8642-2c95763c557b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"
Jul 18 00:41:40.615993 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.615664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8mjc\" (UniqueName: \"kubernetes.io/projected/cd7f0137-a7f4-44ee-8642-2c95763c557b-kube-api-access-r8mjc\") pod \"managed-serviceaccount-addon-agent-74cbcb796b-bm8nl\" (UID: \"cd7f0137-a7f4-44ee-8642-2c95763c557b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"
Jul 18 00:41:40.615993 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.615692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b0afb87-2c01-47c3-aa0d-bc489c69a655-tmp\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.615993 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.615739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7b0afb87-2c01-47c3-aa0d-bc489c69a655-klusterlet-config\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.617143 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.617117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-9gd7s"]
Jul 18 00:41:40.716609 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.716571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9gk\" (UniqueName: \"kubernetes.io/projected/7b0afb87-2c01-47c3-aa0d-bc489c69a655-kube-api-access-tz9gk\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.716796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.716618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd7f0137-a7f4-44ee-8642-2c95763c557b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74cbcb796b-bm8nl\" (UID: \"cd7f0137-a7f4-44ee-8642-2c95763c557b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"
Jul 18 00:41:40.716796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.716647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8mjc\" (UniqueName: \"kubernetes.io/projected/cd7f0137-a7f4-44ee-8642-2c95763c557b-kube-api-access-r8mjc\") pod \"managed-serviceaccount-addon-agent-74cbcb796b-bm8nl\" (UID: \"cd7f0137-a7f4-44ee-8642-2c95763c557b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"
Jul 18 00:41:40.716796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.716671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b0afb87-2c01-47c3-aa0d-bc489c69a655-tmp\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.716796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.716713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7b0afb87-2c01-47c3-aa0d-bc489c69a655-klusterlet-config\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.716796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.716766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9pkj\" (UniqueName: \"kubernetes.io/projected/9f145695-b0d2-48bd-bf15-5beb1a409523-kube-api-access-m9pkj\") pod \"downloads-5bc67dcd46-9gd7s\" (UID: \"9f145695-b0d2-48bd-bf15-5beb1a409523\") " pod="openshift-console/downloads-5bc67dcd46-9gd7s"
Jul 18 00:41:40.717167 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.717140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b0afb87-2c01-47c3-aa0d-bc489c69a655-tmp\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.719498 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.719470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd7f0137-a7f4-44ee-8642-2c95763c557b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74cbcb796b-bm8nl\" (UID: \"cd7f0137-a7f4-44ee-8642-2c95763c557b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"
Jul 18 00:41:40.719608 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.719574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7b0afb87-2c01-47c3-aa0d-bc489c69a655-klusterlet-config\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.730636 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.730608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9gk\" (UniqueName: \"kubernetes.io/projected/7b0afb87-2c01-47c3-aa0d-bc489c69a655-kube-api-access-tz9gk\") pod \"klusterlet-addon-workmgr-85df658b8-mh84t\" (UID: \"7b0afb87-2c01-47c3-aa0d-bc489c69a655\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.730771 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.730718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8mjc\" (UniqueName: \"kubernetes.io/projected/cd7f0137-a7f4-44ee-8642-2c95763c557b-kube-api-access-r8mjc\") pod \"managed-serviceaccount-addon-agent-74cbcb796b-bm8nl\" (UID: \"cd7f0137-a7f4-44ee-8642-2c95763c557b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"
Jul 18 00:41:40.817755 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.817713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9pkj\" (UniqueName: \"kubernetes.io/projected/9f145695-b0d2-48bd-bf15-5beb1a409523-kube-api-access-m9pkj\") pod \"downloads-5bc67dcd46-9gd7s\" (UID: \"9f145695-b0d2-48bd-bf15-5beb1a409523\") " pod="openshift-console/downloads-5bc67dcd46-9gd7s"
Jul 18 00:41:40.820715 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.820686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:40.834897 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.834874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"
Jul 18 00:41:40.835426 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.835400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9pkj\" (UniqueName: \"kubernetes.io/projected/9f145695-b0d2-48bd-bf15-5beb1a409523-kube-api-access-m9pkj\") pod \"downloads-5bc67dcd46-9gd7s\" (UID: \"9f145695-b0d2-48bd-bf15-5beb1a409523\") " pod="openshift-console/downloads-5bc67dcd46-9gd7s"
Jul 18 00:41:40.919161 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:40.919075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-9gd7s"
Jul 18 00:41:44.943013 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:44.942172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl"]
Jul 18 00:41:44.945555 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:41:44.945514    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd7f0137_a7f4_44ee_8642_2c95763c557b.slice/crio-61324350d137abb635effaf34fb38c50349105eabe76abe20c1b381363da54d2 WatchSource:0}: Error finding container 61324350d137abb635effaf34fb38c50349105eabe76abe20c1b381363da54d2: Status 404 returned error can't find the container with id 61324350d137abb635effaf34fb38c50349105eabe76abe20c1b381363da54d2
Jul 18 00:41:44.963399 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:44.963350    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"]
Jul 18 00:41:44.984664 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:44.984547    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-9gd7s"]
Jul 18 00:41:45.637081 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.636503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g5fq8" event={"ID":"2a5b6fba-2c56-4526-9e0f-21ca35cba6f2","Type":"ContainerStarted","Data":"3447e6c64128d01a351d828a21db058d2c96071ac6c1de37795e2dd1d14e6e5a"}
Jul 18 00:41:45.644323 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.644216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t" event={"ID":"7b0afb87-2c01-47c3-aa0d-bc489c69a655","Type":"ContainerStarted","Data":"3a196807fc45f081825c5e54eb8a3dff45d56e4fd1aa2fdffb8d8293570413d3"}
Jul 18 00:41:45.647310 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.647215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4zgt4" event={"ID":"82f6fc6a-2500-4c10-860c-2b2699f81669","Type":"ContainerStarted","Data":"676a9494b0830cdef0dcc8c948b03207c05233f29980b0b3928f77c6ddea096a"}
Jul 18 00:41:45.654024 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.652476    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g5fq8" podStartSLOduration=67.251319052 podStartE2EDuration="1m14.65245626s" totalImagesPullingTime="7.401137208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:45.652019318 +0000 UTC m=+75.971996704" watchObservedRunningTime="2026-07-18 00:41:45.65245626 +0000 UTC m=+75.972433624"
Jul 18 00:41:45.655719 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.654729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzfr6" event={"ID":"ab019bac-1682-4e2c-9167-894dc2ad5cb1","Type":"ContainerStarted","Data":"5f6c5377eb343ff3111bc495954e23c0588746e7321bcd3907afdc21a1e445ad"}
Jul 18 00:41:45.655719 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.654786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzfr6" event={"ID":"ab019bac-1682-4e2c-9167-894dc2ad5cb1","Type":"ContainerStarted","Data":"40fc81269e1c0588bcef484d59650aab0040c9cbe7c970e631ae329f84f0a932"}
Jul 18 00:41:45.655719 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.655549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:45.657171 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.657100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-9gd7s" event={"ID":"9f145695-b0d2-48bd-bf15-5beb1a409523","Type":"ContainerStarted","Data":"f27bc5c95e0add6919d7e89e89e07dcf19a8b8b6e9c0da44c39455e1467b0c9f"}
Jul 18 00:41:45.672774 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.672713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9b8ml" event={"ID":"75ee96b7-bcc7-4f75-bf11-52a674359b51","Type":"ContainerStarted","Data":"4decad99bb06a587e7dbf0b42668e50382a5820f0dead490edcd5bb75cb354b4"}
Jul 18 00:41:45.672957 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.672807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9b8ml" event={"ID":"75ee96b7-bcc7-4f75-bf11-52a674359b51","Type":"ContainerStarted","Data":"313a6b39d5db165fa2b100c710943308f001b572eb274ef0acb848a0b02bbddf"}
Jul 18 00:41:45.675573 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.675511    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jzfr6" podStartSLOduration=33.833414015 podStartE2EDuration="40.675490026s" totalImagesPullingTime="6.842076011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:45.673622377 +0000 UTC m=+75.993599751" watchObservedRunningTime="2026-07-18 00:41:45.675490026 +0000 UTC m=+75.995467411"
Jul 18 00:41:45.681540 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.681508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vv5qj" event={"ID":"509a8c3e-3258-4abb-83a6-ad5bb4722523","Type":"ContainerStarted","Data":"7378be03050c93490a8bfd69caf8769bf4ff3a0e374c8b7b2079220dd9a013dd"}
Jul 18 00:41:45.682132 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.682109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:41:45.685108 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.685078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl" event={"ID":"cd7f0137-a7f4-44ee-8642-2c95763c557b","Type":"ContainerStarted","Data":"61324350d137abb635effaf34fb38c50349105eabe76abe20c1b381363da54d2"}
Jul 18 00:41:45.700068 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.699987    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9b8ml" podStartSLOduration=67.397648983 podStartE2EDuration="1m14.699965368s" totalImagesPullingTime="7.302316385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:45.698785884 +0000 UTC m=+76.018763267" watchObservedRunningTime="2026-07-18 00:41:45.699965368 +0000 UTC m=+76.019942749"
Jul 18 00:41:45.732488 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:45.732421    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vv5qj" podStartSLOduration=67.080790338 podStartE2EDuration="1m14.732401078s" totalImagesPullingTime="7.65161074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:45.731346505 +0000 UTC m=+76.051323890" watchObservedRunningTime="2026-07-18 00:41:45.732401078 +0000 UTC m=+76.052378464"
Jul 18 00:41:52.718999 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:52.718848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl" event={"ID":"cd7f0137-a7f4-44ee-8642-2c95763c557b","Type":"ContainerStarted","Data":"56c0dc9536b86d6ba73fd7690e55926a76d2333e09a2175d14bf2596c3e3cfe5"}
Jul 18 00:41:52.720574 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:52.720541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t" event={"ID":"7b0afb87-2c01-47c3-aa0d-bc489c69a655","Type":"ContainerStarted","Data":"12cebdcc965f0051f6747ec5397297bd5ed19e5ce292ff27944f93feeef4a54a"}
Jul 18 00:41:52.720721 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:52.720697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:52.723179 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:52.723149    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t"
Jul 18 00:41:52.723657 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:52.723632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4zgt4" event={"ID":"82f6fc6a-2500-4c10-860c-2b2699f81669","Type":"ContainerStarted","Data":"0db425f57b6cd52c41be3f6df9a3946ed87e9263fd1f8aa52fdd7576474d3c65"}
Jul 18 00:41:52.734771 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:52.734711    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74cbcb796b-bm8nl" podStartSLOduration=6.040851632 podStartE2EDuration="12.734690353s" totalImagesPullingTime="6.693838721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:52.733236548 +0000 UTC m=+83.053213930" watchObservedRunningTime="2026-07-18 00:41:52.734690353 +0000 UTC m=+83.054667737"
Jul 18 00:41:52.749979 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:52.749916    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4zgt4" podStartSLOduration=18.397157557 podStartE2EDuration="37.74989487s" totalImagesPullingTime="19.352737313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:41:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:52.749007367 +0000 UTC m=+83.068984750" watchObservedRunningTime="2026-07-18 00:41:52.74989487 +0000 UTC m=+83.069872257"
Jul 18 00:41:56.698190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:56.698154    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jzfr6"
Jul 18 00:41:56.716347 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:56.716249    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85df658b8-mh84t" podStartSLOduration=10.022167101 podStartE2EDuration="16.716225396s" totalImagesPullingTime="6.694058295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:52.764476979 +0000 UTC m=+83.084454365" watchObservedRunningTime="2026-07-18 00:41:56.716225396 +0000 UTC m=+87.036202780"
Jul 18 00:41:57.114930 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.114688    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9gbdl"]
Jul 18 00:41:57.119644 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.119552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.126684 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.126487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 18 00:41:57.130937 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.130430    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 18 00:41:57.130937 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.130586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wdls5\""
Jul 18 00:41:57.130937 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.130694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 18 00:41:57.130937 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.130802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 18 00:41:57.130937 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.130442    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 18 00:41:57.133812 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.133783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 18 00:41:57.249863 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-root\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.249863 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.249863 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-textfile\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.249863 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-sys\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.249863 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksft5\" (UniqueName: \"kubernetes.io/projected/1ccc27d1-3918-479e-9da2-ae70606fab11-kube-api-access-ksft5\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.249863 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.250356 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.250356 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ccc27d1-3918-479e-9da2-ae70606fab11-metrics-client-ca\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.250356 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.249949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-wtmp\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351071 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-root\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351263 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351263 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-textfile\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351263 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-root\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351263 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-sys\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351263 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-sys\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351263 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksft5\" (UniqueName: \"kubernetes.io/projected/1ccc27d1-3918-479e-9da2-ae70606fab11-kube-api-access-ksft5\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351584 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351584 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351584 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ccc27d1-3918-479e-9da2-ae70606fab11-metrics-client-ca\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351584 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-wtmp\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351584 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-wtmp\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351584 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.351525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-textfile\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.351584 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:57.351582    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 18 00:41:57.351931 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:57.351657    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls podName:1ccc27d1-3918-479e-9da2-ae70606fab11 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:57.851635642 +0000 UTC m=+88.171613015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls") pod "node-exporter-9gbdl" (UID: "1ccc27d1-3918-479e-9da2-ae70606fab11") : secret "node-exporter-tls" not found
Jul 18 00:41:57.352097 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.352072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.352873 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.352837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ccc27d1-3918-479e-9da2-ae70606fab11-metrics-client-ca\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.354156 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.354132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.362850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.362818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksft5\" (UniqueName: \"kubernetes.io/projected/1ccc27d1-3918-479e-9da2-ae70606fab11-kube-api-access-ksft5\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.683184 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.682836    2563 patch_prober.go:28] interesting pod/image-registry-5f7f84d98c-hf2wv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 18 00:41:57.683184 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.682912    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv" podUID="ea76c39f-22ad-404a-8a10-c0ec2df2697d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 00:41:57.856739 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:57.856684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:57.857276 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:57.856867    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 18 00:41:57.857276 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:41:57.856955    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls podName:1ccc27d1-3918-479e-9da2-ae70606fab11 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:58.85693273 +0000 UTC m=+89.176910104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls") pod "node-exporter-9gbdl" (UID: "1ccc27d1-3918-479e-9da2-ae70606fab11") : secret "node-exporter-tls" not found
Jul 18 00:41:58.867320 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:58.867272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:58.870130 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:58.870100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ccc27d1-3918-479e-9da2-ae70606fab11-node-exporter-tls\") pod \"node-exporter-9gbdl\" (UID: \"1ccc27d1-3918-479e-9da2-ae70606fab11\") " pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:58.934189 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:58.934141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gbdl"
Jul 18 00:41:59.592845 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:41:59.592812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:42:03.544039 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:42:03.544000    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccc27d1_3918_479e_9da2_ae70606fab11.slice/crio-2ef7fa998ae55789ed986542a0f82fa41cec47dfa6ded97bb7788fa05c3f66e0 WatchSource:0}: Error finding container 2ef7fa998ae55789ed986542a0f82fa41cec47dfa6ded97bb7788fa05c3f66e0: Status 404 returned error can't find the container with id 2ef7fa998ae55789ed986542a0f82fa41cec47dfa6ded97bb7788fa05c3f66e0
Jul 18 00:42:03.576257 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.576225    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-679696677-27vqb"]
Jul 18 00:42:03.598141 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.598099    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679696677-27vqb"]
Jul 18 00:42:03.598331 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.598208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.601000 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.600974    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 18 00:42:03.601140 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.601080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 18 00:42:03.601140 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.601097    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 18 00:42:03.602307 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.602288    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 18 00:42:03.602418 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.602288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-56hhc\""
Jul 18 00:42:03.602418 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.602328    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 18 00:42:03.607104 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.607087    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 18 00:42:03.706306 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.706273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-oauth-config\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.706464 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.706325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-trusted-ca-bundle\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.706464 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.706347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc25c\" (UniqueName: \"kubernetes.io/projected/ff2863c4-2276-45c4-b53b-c6adbc2507f6-kube-api-access-qc25c\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.706464 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.706406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-oauth-serving-cert\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.706464 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.706463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-serving-cert\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.706599 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.706479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-config\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.706599 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.706499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-service-ca\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.771134 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.771089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gbdl" event={"ID":"1ccc27d1-3918-479e-9da2-ae70606fab11","Type":"ContainerStarted","Data":"2ef7fa998ae55789ed986542a0f82fa41cec47dfa6ded97bb7788fa05c3f66e0"}
Jul 18 00:42:03.807196 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.807110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-trusted-ca-bundle\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.807196 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.807167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc25c\" (UniqueName: \"kubernetes.io/projected/ff2863c4-2276-45c4-b53b-c6adbc2507f6-kube-api-access-qc25c\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.807421 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.807200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-oauth-serving-cert\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.807421 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.807229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-serving-cert\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.807421 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.807252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-config\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.807421 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.807283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-service-ca\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.807421 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.807336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-oauth-config\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.808007 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.807977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-trusted-ca-bundle\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.808146 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.808016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-config\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.808146 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.808016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-service-ca\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.808307 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.808139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-oauth-serving-cert\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.809771 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.809750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-serving-cert\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.809878 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.809843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-oauth-config\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.815792 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.815769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc25c\" (UniqueName: \"kubernetes.io/projected/ff2863c4-2276-45c4-b53b-c6adbc2507f6-kube-api-access-qc25c\") pod \"console-679696677-27vqb\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") " pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:03.909622 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:03.909552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:04.073914 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:04.073684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679696677-27vqb"]
Jul 18 00:42:04.076571 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:42:04.076536    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2863c4_2276_45c4_b53b_c6adbc2507f6.slice/crio-a319e90003cf46ad54a9c55e1139b38b8df76a4f3c2e08270bf1f069cb447e6f WatchSource:0}: Error finding container a319e90003cf46ad54a9c55e1139b38b8df76a4f3c2e08270bf1f069cb447e6f: Status 404 returned error can't find the container with id a319e90003cf46ad54a9c55e1139b38b8df76a4f3c2e08270bf1f069cb447e6f
Jul 18 00:42:04.776793 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:04.776745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679696677-27vqb" event={"ID":"ff2863c4-2276-45c4-b53b-c6adbc2507f6","Type":"ContainerStarted","Data":"a319e90003cf46ad54a9c55e1139b38b8df76a4f3c2e08270bf1f069cb447e6f"}
Jul 18 00:42:04.778621 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:04.778592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-9gd7s" event={"ID":"9f145695-b0d2-48bd-bf15-5beb1a409523","Type":"ContainerStarted","Data":"3322d7e14937cff7ec211ddc28b57c2ba4ade500c8924b8f157a6d3be22ef74c"}
Jul 18 00:42:04.778838 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:04.778814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-9gd7s"
Jul 18 00:42:04.792192 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:04.792154    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-9gd7s"
Jul 18 00:42:04.793940 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:04.793430    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-9gd7s" podStartSLOduration=5.866247144 podStartE2EDuration="24.793412654s" totalImagesPullingTime="18.92716551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:04.793352031 +0000 UTC m=+95.113329429" watchObservedRunningTime="2026-07-18 00:42:04.793412654 +0000 UTC m=+95.113390037"
Jul 18 00:42:05.786595 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:05.786464    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ccc27d1-3918-479e-9da2-ae70606fab11" containerID="6bcdd8041cb4a04b8a6061d2c9fa4c932ae8d6e4cac44bf31741b69670c62c8f" exitCode=0
Jul 18 00:42:05.787665 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:05.787007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gbdl" event={"ID":"1ccc27d1-3918-479e-9da2-ae70606fab11","Type":"ContainerDied","Data":"6bcdd8041cb4a04b8a6061d2c9fa4c932ae8d6e4cac44bf31741b69670c62c8f"}
Jul 18 00:42:06.793729 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:06.793687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gbdl" event={"ID":"1ccc27d1-3918-479e-9da2-ae70606fab11","Type":"ContainerStarted","Data":"011ab77f94b918a1306c0428b922b601cd1df0f34df552efc4d6e065f7e78186"}
Jul 18 00:42:06.794477 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:06.793755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gbdl" event={"ID":"1ccc27d1-3918-479e-9da2-ae70606fab11","Type":"ContainerStarted","Data":"120934e0e0981fc2dd6b7381dd9e05c02a819c42642644c91c37013cbf1daac1"}
Jul 18 00:42:06.816643 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:06.815979    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9gbdl" podStartSLOduration=8.33854233 podStartE2EDuration="9.815956246s" totalImagesPullingTime="1.373374327s" totalInitContainerRuntime="104.039589ms" isStatefulPod=true podCreationTimestamp="2026-07-18 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:06.814705584 +0000 UTC m=+97.134682981" watchObservedRunningTime="2026-07-18 00:42:06.815956246 +0000 UTC m=+97.135933630"
Jul 18 00:42:08.066009 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:08.065968    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f7f84d98c-hf2wv"]
Jul 18 00:42:08.803772 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:08.803731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679696677-27vqb" event={"ID":"ff2863c4-2276-45c4-b53b-c6adbc2507f6","Type":"ContainerStarted","Data":"b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0"}
Jul 18 00:42:08.823613 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:08.823532    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-679696677-27vqb" podStartSLOduration=1.621418567 podStartE2EDuration="5.823515048s" totalImagesPullingTime="4.202096481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:08.821917297 +0000 UTC m=+99.141894684" watchObservedRunningTime="2026-07-18 00:42:08.823515048 +0000 UTC m=+99.143492431"
Jul 18 00:42:13.909795 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:13.909752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:13.910348 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:13.909849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:13.914726 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:13.914701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:14.834140 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:14.834105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-679696677-27vqb"
Jul 18 00:42:17.698385 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:17.698354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vv5qj"
Jul 18 00:42:33.091437 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.091369    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv" podUID="ea76c39f-22ad-404a-8a10-c0ec2df2697d" containerName="registry" containerID="cri-o://a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c" gracePeriod=30
Jul 18 00:42:33.367332 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.367249    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:42:33.543947 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.543903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-bound-sa-token\") pod \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") "
Jul 18 00:42:33.543947 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.543944    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-image-registry-private-configuration\") pod \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") "
Jul 18 00:42:33.544240 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.543971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") pod \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") "
Jul 18 00:42:33.544240 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.544002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea76c39f-22ad-404a-8a10-c0ec2df2697d-ca-trust-extracted\") pod \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") "
Jul 18 00:42:33.544240 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.544028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh4cr\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-kube-api-access-kh4cr\") pod \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") "
Jul 18 00:42:33.544240 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.544092    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-installation-pull-secrets\") pod \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") "
Jul 18 00:42:33.544240 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.544154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-trusted-ca\") pod \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") "
Jul 18 00:42:33.544240 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.544189    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-certificates\") pod \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\" (UID: \"ea76c39f-22ad-404a-8a10-c0ec2df2697d\") "
Jul 18 00:42:33.544844 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.544808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ea76c39f-22ad-404a-8a10-c0ec2df2697d" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:33.545329 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.545305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ea76c39f-22ad-404a-8a10-c0ec2df2697d" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:42:33.546692 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.546592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ea76c39f-22ad-404a-8a10-c0ec2df2697d" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:42:33.546692 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.546606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ea76c39f-22ad-404a-8a10-c0ec2df2697d" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:42:33.546692 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.546658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ea76c39f-22ad-404a-8a10-c0ec2df2697d" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:42:33.546877 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.546849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ea76c39f-22ad-404a-8a10-c0ec2df2697d" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:42:33.546877 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.546846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-kube-api-access-kh4cr" (OuterVolumeSpecName: "kube-api-access-kh4cr") pod "ea76c39f-22ad-404a-8a10-c0ec2df2697d" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d"). InnerVolumeSpecName "kube-api-access-kh4cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:42:33.553018 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.552987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea76c39f-22ad-404a-8a10-c0ec2df2697d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ea76c39f-22ad-404a-8a10-c0ec2df2697d" (UID: "ea76c39f-22ad-404a-8a10-c0ec2df2697d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:42:33.645550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.645456    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4cr\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-kube-api-access-kh4cr\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:42:33.645550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.645488    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-installation-pull-secrets\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:42:33.645550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.645498    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-trusted-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:42:33.645550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.645507    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-certificates\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:42:33.645550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.645515    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-bound-sa-token\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:42:33.645550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.645526    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea76c39f-22ad-404a-8a10-c0ec2df2697d-image-registry-private-configuration\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:42:33.645550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.645534    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea76c39f-22ad-404a-8a10-c0ec2df2697d-registry-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:42:33.645550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.645544    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea76c39f-22ad-404a-8a10-c0ec2df2697d-ca-trust-extracted\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:42:33.893708 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.893669    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea76c39f-22ad-404a-8a10-c0ec2df2697d" containerID="a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c" exitCode=0
Jul 18 00:42:33.893868 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.893721    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv"
Jul 18 00:42:33.893868 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.893763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv" event={"ID":"ea76c39f-22ad-404a-8a10-c0ec2df2697d","Type":"ContainerDied","Data":"a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c"}
Jul 18 00:42:33.893868 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.893819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7f84d98c-hf2wv" event={"ID":"ea76c39f-22ad-404a-8a10-c0ec2df2697d","Type":"ContainerDied","Data":"0f450df9105bc0f6f0452704fbc4bcc6fcfc1c385cad58f7671f8d062a799552"}
Jul 18 00:42:33.893868 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.893851    2563 scope.go:117] "RemoveContainer" containerID="a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c"
Jul 18 00:42:33.902150 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.902128    2563 scope.go:117] "RemoveContainer" containerID="a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c"
Jul 18 00:42:33.902422 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:42:33.902401    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c\": container with ID starting with a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c not found: ID does not exist" containerID="a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c"
Jul 18 00:42:33.902486 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.902431    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c"} err="failed to get container status \"a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c\": rpc error: code = NotFound desc = could not find container \"a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c\": container with ID starting with a5d09cd14a823a399bd037e80e349173b5c46918f78c7f00f86257b3bc5fea3c not found: ID does not exist"
Jul 18 00:42:33.913436 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.913406    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f7f84d98c-hf2wv"]
Jul 18 00:42:33.916661 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:33.916638    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f7f84d98c-hf2wv"]
Jul 18 00:42:34.174619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:42:34.174528    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea76c39f-22ad-404a-8a10-c0ec2df2697d" path="/var/lib/kubelet/pods/ea76c39f-22ad-404a-8a10-c0ec2df2697d/volumes"
Jul 18 00:43:21.368880 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.368838    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d5ddd96bc-l977r"]
Jul 18 00:43:21.369369 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.369188    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea76c39f-22ad-404a-8a10-c0ec2df2697d" containerName="registry"
Jul 18 00:43:21.369369 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.369201    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea76c39f-22ad-404a-8a10-c0ec2df2697d" containerName="registry"
Jul 18 00:43:21.369369 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.369242    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea76c39f-22ad-404a-8a10-c0ec2df2697d" containerName="registry"
Jul 18 00:43:21.371064 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.371041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.374156 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.374128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 18 00:43:21.374354 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.374156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 18 00:43:21.374448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.374407    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 18 00:43:21.374537 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.374250    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 18 00:43:21.374592 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.374536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t7st4\""
Jul 18 00:43:21.374686 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.374666    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 18 00:43:21.379352 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.379327    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 18 00:43:21.406114 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.406074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d5ddd96bc-l977r"]
Jul 18 00:43:21.457026 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.456982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-secret-telemeter-client\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.457026 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.457028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-serving-certs-ca-bundle\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.457285 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.457104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffbn\" (UniqueName: \"kubernetes.io/projected/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-kube-api-access-7ffbn\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.457285 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.457142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-telemeter-client-tls\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.457285 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.457163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-federate-client-tls\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.457285 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.457227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.457285 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.457275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-metrics-client-ca\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.457467 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.457315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.557814 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.557780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-secret-telemeter-client\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.557916 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.557822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-serving-certs-ca-bundle\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.557916 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.557845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffbn\" (UniqueName: \"kubernetes.io/projected/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-kube-api-access-7ffbn\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.557916 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.557874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-telemeter-client-tls\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.557916 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.557890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-federate-client-tls\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.557916 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.557912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.558180 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.557932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-metrics-client-ca\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.558180 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.558030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.558690 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.558662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-serving-certs-ca-bundle\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.559097 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.559049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-metrics-client-ca\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.559164 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.559097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.560597 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.560574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-telemeter-client-tls\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.560693 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.560622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.560823 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.560805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-federate-client-tls\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.560858 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.560825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-secret-telemeter-client\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.566811 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.566787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffbn\" (UniqueName: \"kubernetes.io/projected/0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54-kube-api-access-7ffbn\") pod \"telemeter-client-d5ddd96bc-l977r\" (UID: \"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54\") " pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.684191 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.684099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r"
Jul 18 00:43:21.820082 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:21.819860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d5ddd96bc-l977r"]
Jul 18 00:43:21.822629 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:43:21.822601    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b76f8fa_dcb3_4dfa_bfee_bec5ac8c7a54.slice/crio-cae0ab245060c3630e18d2ad772e4ae3cb03bdc5a96921772196a1acae777bbc WatchSource:0}: Error finding container cae0ab245060c3630e18d2ad772e4ae3cb03bdc5a96921772196a1acae777bbc: Status 404 returned error can't find the container with id cae0ab245060c3630e18d2ad772e4ae3cb03bdc5a96921772196a1acae777bbc
Jul 18 00:43:22.056093 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:22.056035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r" event={"ID":"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54","Type":"ContainerStarted","Data":"cae0ab245060c3630e18d2ad772e4ae3cb03bdc5a96921772196a1acae777bbc"}
Jul 18 00:43:24.066330 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:24.066242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r" event={"ID":"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54","Type":"ContainerStarted","Data":"85c19a85fee54ea69790cef98a867abf0c9ff47ce99b3ad376dc7cd5d9cc1d09"}
Jul 18 00:43:25.072211 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.072172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r" event={"ID":"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54","Type":"ContainerStarted","Data":"27a36872ba52d5f47b97809936b4a2f643a93d8f6bd7c7a0863159fa27d8bd66"}
Jul 18 00:43:25.072609 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.072231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r" event={"ID":"0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54","Type":"ContainerStarted","Data":"f757a1ca16d7ee2161f42f9620685a8e69ff36ba9a6171946461089380211fb5"}
Jul 18 00:43:25.093332 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.093265    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d5ddd96bc-l977r" podStartSLOduration=1.122148379 podStartE2EDuration="4.093243171s" totalImagesPullingTime="2.971094792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:43:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:43:25.09133128 +0000 UTC m=+175.411308663" watchObservedRunningTime="2026-07-18 00:43:25.093243171 +0000 UTC m=+175.413220557"
Jul 18 00:43:25.849892 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.849853    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64b59494b8-5z2qr"]
Jul 18 00:43:25.852028 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.852009    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:25.863414 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.863384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b59494b8-5z2qr"]
Jul 18 00:43:25.905520 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.905473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-serving-cert\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:25.905520 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.905520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-service-ca\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:25.905766 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.905538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-trusted-ca-bundle\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:25.905766 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.905635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-console-config\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:25.905766 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.905685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjw2n\" (UniqueName: \"kubernetes.io/projected/18242977-4fa4-4650-958e-e7ac64648f45-kube-api-access-xjw2n\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:25.905766 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.905707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-oauth-serving-cert\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:25.905766 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:25.905737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-oauth-config\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.007197 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.007149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-oauth-config\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.007391 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.007211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-serving-cert\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.007391 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.007236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-service-ca\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.007391 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.007252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-trusted-ca-bundle\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.007391 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.007282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-console-config\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.007391 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.007363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjw2n\" (UniqueName: \"kubernetes.io/projected/18242977-4fa4-4650-958e-e7ac64648f45-kube-api-access-xjw2n\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.007639 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.007412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-oauth-serving-cert\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.008231 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.008198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-service-ca\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.008362 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.008246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-oauth-serving-cert\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.008362 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.008273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-console-config\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.008471 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.008402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-trusted-ca-bundle\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.009713 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.009692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-serving-cert\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.009797 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.009713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-oauth-config\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.017073 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.017026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjw2n\" (UniqueName: \"kubernetes.io/projected/18242977-4fa4-4650-958e-e7ac64648f45-kube-api-access-xjw2n\") pod \"console-64b59494b8-5z2qr\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") " pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.160882 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.160777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:26.292102 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:26.292050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b59494b8-5z2qr"]
Jul 18 00:43:26.294824 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:43:26.294797    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18242977_4fa4_4650_958e_e7ac64648f45.slice/crio-485957f16627861bdb8166c81d2ce8b00bd187e35d8e7ef127de2e54ad80767e WatchSource:0}: Error finding container 485957f16627861bdb8166c81d2ce8b00bd187e35d8e7ef127de2e54ad80767e: Status 404 returned error can't find the container with id 485957f16627861bdb8166c81d2ce8b00bd187e35d8e7ef127de2e54ad80767e
Jul 18 00:43:27.083613 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:27.083574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b59494b8-5z2qr" event={"ID":"18242977-4fa4-4650-958e-e7ac64648f45","Type":"ContainerStarted","Data":"40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667"}
Jul 18 00:43:27.083862 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:27.083634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b59494b8-5z2qr" event={"ID":"18242977-4fa4-4650-958e-e7ac64648f45","Type":"ContainerStarted","Data":"485957f16627861bdb8166c81d2ce8b00bd187e35d8e7ef127de2e54ad80767e"}
Jul 18 00:43:27.100748 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:27.100692    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64b59494b8-5z2qr" podStartSLOduration=2.100673983 podStartE2EDuration="2.100673983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:43:27.099406078 +0000 UTC m=+177.419383463" watchObservedRunningTime="2026-07-18 00:43:27.100673983 +0000 UTC m=+177.420651366"
Jul 18 00:43:36.161306 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:36.161267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:36.161702 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:36.161342    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:36.166195 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:36.166170    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:37.124808 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:37.124776    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:43:37.165410 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:43:37.165373    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679696677-27vqb"]
Jul 18 00:44:02.198803 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.198738    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-679696677-27vqb" podUID="ff2863c4-2276-45c4-b53b-c6adbc2507f6" containerName="console" containerID="cri-o://b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0" gracePeriod=15
Jul 18 00:44:02.440738 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.440712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679696677-27vqb"
Jul 18 00:44:02.585830 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.585740    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-service-ca\") pod \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") "
Jul 18 00:44:02.585830 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.585789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc25c\" (UniqueName: \"kubernetes.io/projected/ff2863c4-2276-45c4-b53b-c6adbc2507f6-kube-api-access-qc25c\") pod \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") "
Jul 18 00:44:02.585830 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.585814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-oauth-serving-cert\") pod \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") "
Jul 18 00:44:02.586142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.585842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-oauth-config\") pod \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") "
Jul 18 00:44:02.586142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.585980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-config\") pod \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") "
Jul 18 00:44:02.586142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.586033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-trusted-ca-bundle\") pod \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") "
Jul 18 00:44:02.586142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.586094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-serving-cert\") pod \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\" (UID: \"ff2863c4-2276-45c4-b53b-c6adbc2507f6\") "
Jul 18 00:44:02.586656 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.586387    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ff2863c4-2276-45c4-b53b-c6adbc2507f6" (UID: "ff2863c4-2276-45c4-b53b-c6adbc2507f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:44:02.586656 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.586397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-service-ca" (OuterVolumeSpecName: "service-ca") pod "ff2863c4-2276-45c4-b53b-c6adbc2507f6" (UID: "ff2863c4-2276-45c4-b53b-c6adbc2507f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:44:02.586656 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.586450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-config" (OuterVolumeSpecName: "console-config") pod "ff2863c4-2276-45c4-b53b-c6adbc2507f6" (UID: "ff2863c4-2276-45c4-b53b-c6adbc2507f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:44:02.586656 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.586474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ff2863c4-2276-45c4-b53b-c6adbc2507f6" (UID: "ff2863c4-2276-45c4-b53b-c6adbc2507f6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:44:02.588398 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.588370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ff2863c4-2276-45c4-b53b-c6adbc2507f6" (UID: "ff2863c4-2276-45c4-b53b-c6adbc2507f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:44:02.588398 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.588385    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ff2863c4-2276-45c4-b53b-c6adbc2507f6" (UID: "ff2863c4-2276-45c4-b53b-c6adbc2507f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:44:02.588398 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.588381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2863c4-2276-45c4-b53b-c6adbc2507f6-kube-api-access-qc25c" (OuterVolumeSpecName: "kube-api-access-qc25c") pod "ff2863c4-2276-45c4-b53b-c6adbc2507f6" (UID: "ff2863c4-2276-45c4-b53b-c6adbc2507f6"). InnerVolumeSpecName "kube-api-access-qc25c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:44:02.687277 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.687241    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:44:02.687277 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.687270    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc25c\" (UniqueName: \"kubernetes.io/projected/ff2863c4-2276-45c4-b53b-c6adbc2507f6-kube-api-access-qc25c\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:44:02.687277 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.687280    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-oauth-serving-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:44:02.687506 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.687292    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-oauth-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:44:02.687506 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.687301    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:44:02.687506 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.687309    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff2863c4-2276-45c4-b53b-c6adbc2507f6-trusted-ca-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:44:02.687506 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:02.687317    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff2863c4-2276-45c4-b53b-c6adbc2507f6-console-serving-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:44:03.210811 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.210777    2563 generic.go:356] "Generic (PLEG): container finished" podID="ff2863c4-2276-45c4-b53b-c6adbc2507f6" containerID="b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0" exitCode=0
Jul 18 00:44:03.211233 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.210835    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679696677-27vqb"
Jul 18 00:44:03.211233 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.210857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679696677-27vqb" event={"ID":"ff2863c4-2276-45c4-b53b-c6adbc2507f6","Type":"ContainerDied","Data":"b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0"}
Jul 18 00:44:03.211233 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.210899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679696677-27vqb" event={"ID":"ff2863c4-2276-45c4-b53b-c6adbc2507f6","Type":"ContainerDied","Data":"a319e90003cf46ad54a9c55e1139b38b8df76a4f3c2e08270bf1f069cb447e6f"}
Jul 18 00:44:03.211233 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.210918    2563 scope.go:117] "RemoveContainer" containerID="b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0"
Jul 18 00:44:03.219587 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.219570    2563 scope.go:117] "RemoveContainer" containerID="b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0"
Jul 18 00:44:03.219838 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:44:03.219819    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0\": container with ID starting with b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0 not found: ID does not exist" containerID="b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0"
Jul 18 00:44:03.219882 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.219847    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0"} err="failed to get container status \"b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0\": rpc error: code = NotFound desc = could not find container \"b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0\": container with ID starting with b156376da0f12dc3856109be7fc71028a91f38bec4bdacb3946a92ea1e5b7cb0 not found: ID does not exist"
Jul 18 00:44:03.230439 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.230413    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679696677-27vqb"]
Jul 18 00:44:03.235982 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:03.235959    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-679696677-27vqb"]
Jul 18 00:44:04.178224 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:04.178188    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff2863c4-2276-45c4-b53b-c6adbc2507f6" path="/var/lib/kubelet/pods/ff2863c4-2276-45c4-b53b-c6adbc2507f6/volumes"
Jul 18 00:44:38.442165 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.442125    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b95cf4c4-cfxsb"]
Jul 18 00:44:38.442903 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.442393    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2863c4-2276-45c4-b53b-c6adbc2507f6" containerName="console"
Jul 18 00:44:38.442903 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.442404    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2863c4-2276-45c4-b53b-c6adbc2507f6" containerName="console"
Jul 18 00:44:38.442903 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.442462    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2863c4-2276-45c4-b53b-c6adbc2507f6" containerName="console"
Jul 18 00:44:38.446418 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.446395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.450514 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.450491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzmq\" (UniqueName: \"kubernetes.io/projected/9af95e73-31d8-4956-a69d-10a389a88844-kube-api-access-2qzmq\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.450653 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.450524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-oauth-serving-cert\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.450653 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.450542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-console-config\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.450778 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.450623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-serving-cert\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.451757 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.450695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-service-ca\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.451856 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.451792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-trusted-ca-bundle\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.451904 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.451853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-oauth-config\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.456571 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.456550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b95cf4c4-cfxsb"]
Jul 18 00:44:38.552687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.552646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-oauth-config\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.552687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.552691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzmq\" (UniqueName: \"kubernetes.io/projected/9af95e73-31d8-4956-a69d-10a389a88844-kube-api-access-2qzmq\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.552929 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.552813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-oauth-serving-cert\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.552929 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.552854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-console-config\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.552929 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.552893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-serving-cert\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.552929 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.552917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-service-ca\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.553178 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.552951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-trusted-ca-bundle\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.553640 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.553615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-service-ca\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.553745 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.553639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-oauth-serving-cert\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.553784 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.553735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-console-config\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.553890 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.553873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-trusted-ca-bundle\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.555156 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.555129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-oauth-config\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.555292 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.555276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-serving-cert\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.560486 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.560466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzmq\" (UniqueName: \"kubernetes.io/projected/9af95e73-31d8-4956-a69d-10a389a88844-kube-api-access-2qzmq\") pod \"console-7b95cf4c4-cfxsb\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") " pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.757177 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.757121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:38.886622 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:38.886588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b95cf4c4-cfxsb"]
Jul 18 00:44:38.890340 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:44:38.890309    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af95e73_31d8_4956_a69d_10a389a88844.slice/crio-ee98ea2a8b6198fb130e347b4aad934db31e3e79baeda48ca23b90391fffafd6 WatchSource:0}: Error finding container ee98ea2a8b6198fb130e347b4aad934db31e3e79baeda48ca23b90391fffafd6: Status 404 returned error can't find the container with id ee98ea2a8b6198fb130e347b4aad934db31e3e79baeda48ca23b90391fffafd6
Jul 18 00:44:39.337194 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:39.337147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b95cf4c4-cfxsb" event={"ID":"9af95e73-31d8-4956-a69d-10a389a88844","Type":"ContainerStarted","Data":"1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f"}
Jul 18 00:44:39.337383 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:39.337212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b95cf4c4-cfxsb" event={"ID":"9af95e73-31d8-4956-a69d-10a389a88844","Type":"ContainerStarted","Data":"ee98ea2a8b6198fb130e347b4aad934db31e3e79baeda48ca23b90391fffafd6"}
Jul 18 00:44:39.354835 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:39.354780    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b95cf4c4-cfxsb" podStartSLOduration=1.354762662 podStartE2EDuration="1.354762662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:44:39.353669234 +0000 UTC m=+249.673646629" watchObservedRunningTime="2026-07-18 00:44:39.354762662 +0000 UTC m=+249.674740044"
Jul 18 00:44:48.758237 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:48.758190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:48.758703 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:48.758268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:48.763234 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:48.763208    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:49.376904 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:49.376873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:44:49.417425 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:44:49.417394    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64b59494b8-5z2qr"]
Jul 18 00:45:14.444199 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.444140    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64b59494b8-5z2qr" podUID="18242977-4fa4-4650-958e-e7ac64648f45" containerName="console" containerID="cri-o://40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667" gracePeriod=15
Jul 18 00:45:14.688372 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.688340    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:45:14.823222 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823188    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjw2n\" (UniqueName: \"kubernetes.io/projected/18242977-4fa4-4650-958e-e7ac64648f45-kube-api-access-xjw2n\") pod \"18242977-4fa4-4650-958e-e7ac64648f45\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") "
Jul 18 00:45:14.823222 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-service-ca\") pod \"18242977-4fa4-4650-958e-e7ac64648f45\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") "
Jul 18 00:45:14.823476 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-oauth-config\") pod \"18242977-4fa4-4650-958e-e7ac64648f45\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") "
Jul 18 00:45:14.823476 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823279    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-oauth-serving-cert\") pod \"18242977-4fa4-4650-958e-e7ac64648f45\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") "
Jul 18 00:45:14.823476 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823305    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-console-config\") pod \"18242977-4fa4-4650-958e-e7ac64648f45\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") "
Jul 18 00:45:14.823476 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823403    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-serving-cert\") pod \"18242977-4fa4-4650-958e-e7ac64648f45\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") "
Jul 18 00:45:14.823476 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823465    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-trusted-ca-bundle\") pod \"18242977-4fa4-4650-958e-e7ac64648f45\" (UID: \"18242977-4fa4-4650-958e-e7ac64648f45\") "
Jul 18 00:45:14.823731 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "18242977-4fa4-4650-958e-e7ac64648f45" (UID: "18242977-4fa4-4650-958e-e7ac64648f45"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:45:14.823790 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-console-config" (OuterVolumeSpecName: "console-config") pod "18242977-4fa4-4650-958e-e7ac64648f45" (UID: "18242977-4fa4-4650-958e-e7ac64648f45"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:45:14.823933 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.823896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "18242977-4fa4-4650-958e-e7ac64648f45" (UID: "18242977-4fa4-4650-958e-e7ac64648f45"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:45:14.824156 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.824133    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-service-ca" (OuterVolumeSpecName: "service-ca") pod "18242977-4fa4-4650-958e-e7ac64648f45" (UID: "18242977-4fa4-4650-958e-e7ac64648f45"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:45:14.825491 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.825457    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "18242977-4fa4-4650-958e-e7ac64648f45" (UID: "18242977-4fa4-4650-958e-e7ac64648f45"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:45:14.825491 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.825468    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "18242977-4fa4-4650-958e-e7ac64648f45" (UID: "18242977-4fa4-4650-958e-e7ac64648f45"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:45:14.825614 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.825537    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18242977-4fa4-4650-958e-e7ac64648f45-kube-api-access-xjw2n" (OuterVolumeSpecName: "kube-api-access-xjw2n") pod "18242977-4fa4-4650-958e-e7ac64648f45" (UID: "18242977-4fa4-4650-958e-e7ac64648f45"). InnerVolumeSpecName "kube-api-access-xjw2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:45:14.924046 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.923987    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-oauth-serving-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:14.924046 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.924037    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-console-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:14.924046 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.924046    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-serving-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:14.924046 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.924078    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-trusted-ca-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:14.924046 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.924088    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjw2n\" (UniqueName: \"kubernetes.io/projected/18242977-4fa4-4650-958e-e7ac64648f45-kube-api-access-xjw2n\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:14.924364 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.924096    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18242977-4fa4-4650-958e-e7ac64648f45-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:14.924364 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:14.924105    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18242977-4fa4-4650-958e-e7ac64648f45-console-oauth-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:15.462670 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.462634    2563 generic.go:356] "Generic (PLEG): container finished" podID="18242977-4fa4-4650-958e-e7ac64648f45" containerID="40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667" exitCode=0
Jul 18 00:45:15.463147 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.462676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b59494b8-5z2qr" event={"ID":"18242977-4fa4-4650-958e-e7ac64648f45","Type":"ContainerDied","Data":"40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667"}
Jul 18 00:45:15.463147 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.462684    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b59494b8-5z2qr"
Jul 18 00:45:15.463147 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.462726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b59494b8-5z2qr" event={"ID":"18242977-4fa4-4650-958e-e7ac64648f45","Type":"ContainerDied","Data":"485957f16627861bdb8166c81d2ce8b00bd187e35d8e7ef127de2e54ad80767e"}
Jul 18 00:45:15.463147 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.462753    2563 scope.go:117] "RemoveContainer" containerID="40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667"
Jul 18 00:45:15.471521 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.471501    2563 scope.go:117] "RemoveContainer" containerID="40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667"
Jul 18 00:45:15.471792 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:45:15.471772    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667\": container with ID starting with 40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667 not found: ID does not exist" containerID="40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667"
Jul 18 00:45:15.471849 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.471803    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667"} err="failed to get container status \"40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667\": rpc error: code = NotFound desc = could not find container \"40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667\": container with ID starting with 40198cb375ab0f679efaa702932faf34c3b8cff843e675543c1848dc967e1667 not found: ID does not exist"
Jul 18 00:45:15.482714 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.482685    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64b59494b8-5z2qr"]
Jul 18 00:45:15.485723 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:15.485696    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64b59494b8-5z2qr"]
Jul 18 00:45:16.173778 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:16.173741    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18242977-4fa4-4650-958e-e7ac64648f45" path="/var/lib/kubelet/pods/18242977-4fa4-4650-958e-e7ac64648f45/volumes"
Jul 18 00:45:22.742908 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.742871    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"]
Jul 18 00:45:22.743387 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.743132    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18242977-4fa4-4650-958e-e7ac64648f45" containerName="console"
Jul 18 00:45:22.743387 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.743144    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="18242977-4fa4-4650-958e-e7ac64648f45" containerName="console"
Jul 18 00:45:22.743387 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.743195    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="18242977-4fa4-4650-958e-e7ac64648f45" containerName="console"
Jul 18 00:45:22.746023 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.746005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.748714 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.748689    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 18 00:45:22.748714 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.748694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 18 00:45:22.749540 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.749526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5w4gn\""
Jul 18 00:45:22.752759 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.752729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"]
Jul 18 00:45:22.883364 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.883319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.883364 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.883364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.883603 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.883387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/03d69990-9424-40ed-8eab-2e847d981134-kube-api-access-5nldx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.984485 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.984444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.984485 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.984492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.984730 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.984525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/03d69990-9424-40ed-8eab-2e847d981134-kube-api-access-5nldx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.984865 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.984844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.984930 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.984861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:22.992785 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:22.992760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/03d69990-9424-40ed-8eab-2e847d981134-kube-api-access-5nldx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:23.055610 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:23.055522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:23.182914 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:23.182787    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"]
Jul 18 00:45:23.185587 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:45:23.185563    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d69990_9424_40ed_8eab_2e847d981134.slice/crio-8d1ddcd43f7e85656d22a23b86887b0ab5f8e67feb5936caedd78993c2fd8019 WatchSource:0}: Error finding container 8d1ddcd43f7e85656d22a23b86887b0ab5f8e67feb5936caedd78993c2fd8019: Status 404 returned error can't find the container with id 8d1ddcd43f7e85656d22a23b86887b0ab5f8e67feb5936caedd78993c2fd8019
Jul 18 00:45:23.492656 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:23.492620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx" event={"ID":"03d69990-9424-40ed-8eab-2e847d981134","Type":"ContainerStarted","Data":"8d1ddcd43f7e85656d22a23b86887b0ab5f8e67feb5936caedd78993c2fd8019"}
Jul 18 00:45:30.063246 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:30.063222    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 18 00:45:30.520350 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:30.520307    2563 generic.go:356] "Generic (PLEG): container finished" podID="03d69990-9424-40ed-8eab-2e847d981134" containerID="dce344e01d8a00d95a3c71318852a7bd677a930e5dc534c066364723aaf0e4b8" exitCode=0
Jul 18 00:45:30.522474 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:30.520365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx" event={"ID":"03d69990-9424-40ed-8eab-2e847d981134","Type":"ContainerDied","Data":"dce344e01d8a00d95a3c71318852a7bd677a930e5dc534c066364723aaf0e4b8"}
Jul 18 00:45:33.533402 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:33.533363    2563 generic.go:356] "Generic (PLEG): container finished" podID="03d69990-9424-40ed-8eab-2e847d981134" containerID="2e82fc2b56dbe7465a0ace82f106e8a08f12c2e97f1cd6a6e310aaeb385584b5" exitCode=0
Jul 18 00:45:33.533785 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:33.533448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx" event={"ID":"03d69990-9424-40ed-8eab-2e847d981134","Type":"ContainerDied","Data":"2e82fc2b56dbe7465a0ace82f106e8a08f12c2e97f1cd6a6e310aaeb385584b5"}
Jul 18 00:45:33.534421 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:33.534404    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 00:45:40.564866 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:40.564829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx" event={"ID":"03d69990-9424-40ed-8eab-2e847d981134","Type":"ContainerStarted","Data":"ed619a63c6d96e23c3434a84db9fde4dd0a2627b55039a3dd63bc8e97155e95f"}
Jul 18 00:45:40.583797 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:40.583738    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx" podStartSLOduration=2.800038163 podStartE2EDuration="18.583722892s" totalImagesPullingTime="15.783684729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:45:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:45:40.582386447 +0000 UTC m=+310.902363832" watchObservedRunningTime="2026-07-18 00:45:40.583722892 +0000 UTC m=+310.903700274"
Jul 18 00:45:41.570598 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:41.570559    2563 generic.go:356] "Generic (PLEG): container finished" podID="03d69990-9424-40ed-8eab-2e847d981134" containerID="ed619a63c6d96e23c3434a84db9fde4dd0a2627b55039a3dd63bc8e97155e95f" exitCode=0
Jul 18 00:45:41.570598 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:41.570602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx" event={"ID":"03d69990-9424-40ed-8eab-2e847d981134","Type":"ContainerDied","Data":"ed619a63c6d96e23c3434a84db9fde4dd0a2627b55039a3dd63bc8e97155e95f"}
Jul 18 00:45:42.700626 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.700597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:42.850594 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.850492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-bundle\") pod \"03d69990-9424-40ed-8eab-2e847d981134\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") "
Jul 18 00:45:42.850594 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.850563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-util\") pod \"03d69990-9424-40ed-8eab-2e847d981134\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") "
Jul 18 00:45:42.850830 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.850601    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/03d69990-9424-40ed-8eab-2e847d981134-kube-api-access-5nldx\") pod \"03d69990-9424-40ed-8eab-2e847d981134\" (UID: \"03d69990-9424-40ed-8eab-2e847d981134\") "
Jul 18 00:45:42.851269 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.851244    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-bundle" (OuterVolumeSpecName: "bundle") pod "03d69990-9424-40ed-8eab-2e847d981134" (UID: "03d69990-9424-40ed-8eab-2e847d981134"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:45:42.852777 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.852757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d69990-9424-40ed-8eab-2e847d981134-kube-api-access-5nldx" (OuterVolumeSpecName: "kube-api-access-5nldx") pod "03d69990-9424-40ed-8eab-2e847d981134" (UID: "03d69990-9424-40ed-8eab-2e847d981134"). InnerVolumeSpecName "kube-api-access-5nldx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:45:42.856172 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.856143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-util" (OuterVolumeSpecName: "util") pod "03d69990-9424-40ed-8eab-2e847d981134" (UID: "03d69990-9424-40ed-8eab-2e847d981134"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:45:42.951350 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.951299    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:42.951350 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.951347    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d69990-9424-40ed-8eab-2e847d981134-util\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:42.951350 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:42.951356    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/03d69990-9424-40ed-8eab-2e847d981134-kube-api-access-5nldx\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:45:43.579637 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:43.579600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx" event={"ID":"03d69990-9424-40ed-8eab-2e847d981134","Type":"ContainerDied","Data":"8d1ddcd43f7e85656d22a23b86887b0ab5f8e67feb5936caedd78993c2fd8019"}
Jul 18 00:45:43.579809 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:43.579641    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pvrrx"
Jul 18 00:45:43.579809 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:43.579647    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d1ddcd43f7e85656d22a23b86887b0ab5f8e67feb5936caedd78993c2fd8019"
Jul 18 00:45:50.686964 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.686928    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"]
Jul 18 00:45:50.687448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.687208    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d69990-9424-40ed-8eab-2e847d981134" containerName="pull"
Jul 18 00:45:50.687448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.687221    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d69990-9424-40ed-8eab-2e847d981134" containerName="pull"
Jul 18 00:45:50.687448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.687228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d69990-9424-40ed-8eab-2e847d981134" containerName="extract"
Jul 18 00:45:50.687448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.687234    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d69990-9424-40ed-8eab-2e847d981134" containerName="extract"
Jul 18 00:45:50.687448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.687245    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d69990-9424-40ed-8eab-2e847d981134" containerName="util"
Jul 18 00:45:50.687448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.687250    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d69990-9424-40ed-8eab-2e847d981134" containerName="util"
Jul 18 00:45:50.687448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.687292    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d69990-9424-40ed-8eab-2e847d981134" containerName="extract"
Jul 18 00:45:50.729669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.729632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"]
Jul 18 00:45:50.729825 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.729749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"
Jul 18 00:45:50.732941 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.732914    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 18 00:45:50.732941 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.732938    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 18 00:45:50.733161 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.733041    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jjd7j\""
Jul 18 00:45:50.911702 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.911651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69d5d\" (UniqueName: \"kubernetes.io/projected/2423abd4-a66a-4b34-84c8-aa6ec2b20d73-kube-api-access-69d5d\") pod \"cert-manager-operator-controller-manager-55c444bf86-jqbgz\" (UID: \"2423abd4-a66a-4b34-84c8-aa6ec2b20d73\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"
Jul 18 00:45:50.911889 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:50.911728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2423abd4-a66a-4b34-84c8-aa6ec2b20d73-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jqbgz\" (UID: \"2423abd4-a66a-4b34-84c8-aa6ec2b20d73\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"
Jul 18 00:45:51.012589 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:51.012550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69d5d\" (UniqueName: \"kubernetes.io/projected/2423abd4-a66a-4b34-84c8-aa6ec2b20d73-kube-api-access-69d5d\") pod \"cert-manager-operator-controller-manager-55c444bf86-jqbgz\" (UID: \"2423abd4-a66a-4b34-84c8-aa6ec2b20d73\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"
Jul 18 00:45:51.012786 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:51.012611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2423abd4-a66a-4b34-84c8-aa6ec2b20d73-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jqbgz\" (UID: \"2423abd4-a66a-4b34-84c8-aa6ec2b20d73\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"
Jul 18 00:45:51.012935 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:51.012920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2423abd4-a66a-4b34-84c8-aa6ec2b20d73-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jqbgz\" (UID: \"2423abd4-a66a-4b34-84c8-aa6ec2b20d73\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"
Jul 18 00:45:51.020838 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:51.020810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69d5d\" (UniqueName: \"kubernetes.io/projected/2423abd4-a66a-4b34-84c8-aa6ec2b20d73-kube-api-access-69d5d\") pod \"cert-manager-operator-controller-manager-55c444bf86-jqbgz\" (UID: \"2423abd4-a66a-4b34-84c8-aa6ec2b20d73\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"
Jul 18 00:45:51.041655 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:51.041627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"
Jul 18 00:45:51.190446 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:51.190411    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz"]
Jul 18 00:45:51.193879 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:45:51.193849    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2423abd4_a66a_4b34_84c8_aa6ec2b20d73.slice/crio-14ca580c43c62678ce057232c6c869bf0770b51e384454aad8a9428171dcf398 WatchSource:0}: Error finding container 14ca580c43c62678ce057232c6c869bf0770b51e384454aad8a9428171dcf398: Status 404 returned error can't find the container with id 14ca580c43c62678ce057232c6c869bf0770b51e384454aad8a9428171dcf398
Jul 18 00:45:51.608588 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:51.608553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz" event={"ID":"2423abd4-a66a-4b34-84c8-aa6ec2b20d73","Type":"ContainerStarted","Data":"14ca580c43c62678ce057232c6c869bf0770b51e384454aad8a9428171dcf398"}
Jul 18 00:45:53.618756 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:53.618722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz" event={"ID":"2423abd4-a66a-4b34-84c8-aa6ec2b20d73","Type":"ContainerStarted","Data":"c7beca6039b388184bfdaa390e32916dd83ec2978eed7e32135b6485a5edef90"}
Jul 18 00:45:53.640051 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:53.639992    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jqbgz" podStartSLOduration=1.653980182 podStartE2EDuration="3.639974302s" totalImagesPullingTime="1.98599412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:45:53.6375697 +0000 UTC m=+323.957547082" watchObservedRunningTime="2026-07-18 00:45:53.639974302 +0000 UTC m=+323.959951685"
Jul 18 00:45:54.792912 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.792876    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"]
Jul 18 00:45:54.795589 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.795566    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.797963 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.797938    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5w4gn\""
Jul 18 00:45:54.797963 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.797957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 18 00:45:54.798145 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.797937    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 18 00:45:54.803954 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.803928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"]
Jul 18 00:45:54.841925 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.841892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.841925 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.841928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8bz\" (UniqueName: \"kubernetes.io/projected/1c0f3da3-413e-4957-b762-81714396b8e4-kube-api-access-td8bz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.842154 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.841981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.942374 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.942331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.942374 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.942376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8bz\" (UniqueName: \"kubernetes.io/projected/1c0f3da3-413e-4957-b762-81714396b8e4-kube-api-access-td8bz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.942622 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.942407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.942738 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.942720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.942852 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.942834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:54.954499 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:54.954474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8bz\" (UniqueName: \"kubernetes.io/projected/1c0f3da3-413e-4957-b762-81714396b8e4-kube-api-access-td8bz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:55.105647 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:55.105559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:45:55.240518 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:55.240462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"]
Jul 18 00:45:55.243135 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:45:55.243111    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0f3da3_413e_4957_b762_81714396b8e4.slice/crio-c6d80dbc3ef2a7049712ade9d6c774771649061a1c997d6e7468401c6f466774 WatchSource:0}: Error finding container c6d80dbc3ef2a7049712ade9d6c774771649061a1c997d6e7468401c6f466774: Status 404 returned error can't find the container with id c6d80dbc3ef2a7049712ade9d6c774771649061a1c997d6e7468401c6f466774
Jul 18 00:45:55.628400 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:55.628363    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c0f3da3-413e-4957-b762-81714396b8e4" containerID="8c2e94496f182815729fc91293bfb96d62a6376f666afc6d305e2aa0aa4a8d45" exitCode=0
Jul 18 00:45:55.628582 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:55.628442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g" event={"ID":"1c0f3da3-413e-4957-b762-81714396b8e4","Type":"ContainerDied","Data":"8c2e94496f182815729fc91293bfb96d62a6376f666afc6d305e2aa0aa4a8d45"}
Jul 18 00:45:55.628582 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:55.628492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g" event={"ID":"1c0f3da3-413e-4957-b762-81714396b8e4","Type":"ContainerStarted","Data":"c6d80dbc3ef2a7049712ade9d6c774771649061a1c997d6e7468401c6f466774"}
Jul 18 00:45:57.638050 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.638001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g" event={"ID":"1c0f3da3-413e-4957-b762-81714396b8e4","Type":"ContainerStarted","Data":"6f61d3e2fc786db9c85ac653d0b4354da5a10e404db69c94d82cdcb74828339b"}
Jul 18 00:45:57.841737 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.841656    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"]
Jul 18 00:45:57.844775 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.844755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"
Jul 18 00:45:57.847534 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.847506    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 18 00:45:57.847658 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.847517    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4vd4s\""
Jul 18 00:45:57.847658 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.847518    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 18 00:45:57.856253 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.856230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"]
Jul 18 00:45:57.866562 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.866537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxqv\" (UniqueName: \"kubernetes.io/projected/e83b5550-ca58-45f1-a6e6-7b5335994d5f-kube-api-access-dtxqv\") pod \"cert-manager-cainjector-559cdd8c87-7fc4z\" (UID: \"e83b5550-ca58-45f1-a6e6-7b5335994d5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"
Jul 18 00:45:57.866687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.866604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e83b5550-ca58-45f1-a6e6-7b5335994d5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7fc4z\" (UID: \"e83b5550-ca58-45f1-a6e6-7b5335994d5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"
Jul 18 00:45:57.967894 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.967859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e83b5550-ca58-45f1-a6e6-7b5335994d5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7fc4z\" (UID: \"e83b5550-ca58-45f1-a6e6-7b5335994d5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"
Jul 18 00:45:57.968127 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.967903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxqv\" (UniqueName: \"kubernetes.io/projected/e83b5550-ca58-45f1-a6e6-7b5335994d5f-kube-api-access-dtxqv\") pod \"cert-manager-cainjector-559cdd8c87-7fc4z\" (UID: \"e83b5550-ca58-45f1-a6e6-7b5335994d5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"
Jul 18 00:45:57.978645 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.978608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e83b5550-ca58-45f1-a6e6-7b5335994d5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7fc4z\" (UID: \"e83b5550-ca58-45f1-a6e6-7b5335994d5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"
Jul 18 00:45:57.978749 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:57.978696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxqv\" (UniqueName: \"kubernetes.io/projected/e83b5550-ca58-45f1-a6e6-7b5335994d5f-kube-api-access-dtxqv\") pod \"cert-manager-cainjector-559cdd8c87-7fc4z\" (UID: \"e83b5550-ca58-45f1-a6e6-7b5335994d5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"
Jul 18 00:45:58.154373 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:58.154285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"
Jul 18 00:45:58.282737 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:58.282702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z"]
Jul 18 00:45:58.285399 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:45:58.285370    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode83b5550_ca58_45f1_a6e6_7b5335994d5f.slice/crio-543428097f7f57a723abc6f260e87021b3ebf4f013af69a557364c75a36e3f08 WatchSource:0}: Error finding container 543428097f7f57a723abc6f260e87021b3ebf4f013af69a557364c75a36e3f08: Status 404 returned error can't find the container with id 543428097f7f57a723abc6f260e87021b3ebf4f013af69a557364c75a36e3f08
Jul 18 00:45:58.644343 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:58.644305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z" event={"ID":"e83b5550-ca58-45f1-a6e6-7b5335994d5f","Type":"ContainerStarted","Data":"543428097f7f57a723abc6f260e87021b3ebf4f013af69a557364c75a36e3f08"}
Jul 18 00:45:58.646090 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:58.646039    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c0f3da3-413e-4957-b762-81714396b8e4" containerID="6f61d3e2fc786db9c85ac653d0b4354da5a10e404db69c94d82cdcb74828339b" exitCode=0
Jul 18 00:45:58.646194 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:58.646087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g" event={"ID":"1c0f3da3-413e-4957-b762-81714396b8e4","Type":"ContainerDied","Data":"6f61d3e2fc786db9c85ac653d0b4354da5a10e404db69c94d82cdcb74828339b"}
Jul 18 00:45:59.653792 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:59.653750    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c0f3da3-413e-4957-b762-81714396b8e4" containerID="59f0b511a974eeb6253ea38654f955637b36c9602096c8a00a966771e60ca51e" exitCode=0
Jul 18 00:45:59.654235 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:45:59.653846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g" event={"ID":"1c0f3da3-413e-4957-b762-81714396b8e4","Type":"ContainerDied","Data":"59f0b511a974eeb6253ea38654f955637b36c9602096c8a00a966771e60ca51e"}
Jul 18 00:46:00.853505 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.853478    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:46:00.887047 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.887014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-bundle\") pod \"1c0f3da3-413e-4957-b762-81714396b8e4\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") "
Jul 18 00:46:00.887199 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.887087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-util\") pod \"1c0f3da3-413e-4957-b762-81714396b8e4\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") "
Jul 18 00:46:00.887199 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.887137    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td8bz\" (UniqueName: \"kubernetes.io/projected/1c0f3da3-413e-4957-b762-81714396b8e4-kube-api-access-td8bz\") pod \"1c0f3da3-413e-4957-b762-81714396b8e4\" (UID: \"1c0f3da3-413e-4957-b762-81714396b8e4\") "
Jul 18 00:46:00.887448 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.887421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-bundle" (OuterVolumeSpecName: "bundle") pod "1c0f3da3-413e-4957-b762-81714396b8e4" (UID: "1c0f3da3-413e-4957-b762-81714396b8e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:46:00.889492 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.889462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0f3da3-413e-4957-b762-81714396b8e4-kube-api-access-td8bz" (OuterVolumeSpecName: "kube-api-access-td8bz") pod "1c0f3da3-413e-4957-b762-81714396b8e4" (UID: "1c0f3da3-413e-4957-b762-81714396b8e4"). InnerVolumeSpecName "kube-api-access-td8bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:46:00.893645 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.893615    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-util" (OuterVolumeSpecName: "util") pod "1c0f3da3-413e-4957-b762-81714396b8e4" (UID: "1c0f3da3-413e-4957-b762-81714396b8e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:46:00.988255 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.988222    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:46:00.988255 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.988258    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0f3da3-413e-4957-b762-81714396b8e4-util\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:46:00.988413 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:00.988276    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td8bz\" (UniqueName: \"kubernetes.io/projected/1c0f3da3-413e-4957-b762-81714396b8e4-kube-api-access-td8bz\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:46:01.664399 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:01.664367    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g"
Jul 18 00:46:01.664399 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:01.664392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4vv9g" event={"ID":"1c0f3da3-413e-4957-b762-81714396b8e4","Type":"ContainerDied","Data":"c6d80dbc3ef2a7049712ade9d6c774771649061a1c997d6e7468401c6f466774"}
Jul 18 00:46:01.664712 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:01.664443    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d80dbc3ef2a7049712ade9d6c774771649061a1c997d6e7468401c6f466774"
Jul 18 00:46:01.665881 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:01.665846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z" event={"ID":"e83b5550-ca58-45f1-a6e6-7b5335994d5f","Type":"ContainerStarted","Data":"21d8e87c65d68b454f5bd4fbd4ca70cd669a2baed95bb67da10f02b506158175"}
Jul 18 00:46:01.688037 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:01.687981    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7fc4z" podStartSLOduration=2.070582372 podStartE2EDuration="4.687960882s" totalImagesPullingTime="2.61737851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:01.687727924 +0000 UTC m=+332.007705340" watchObservedRunningTime="2026-07-18 00:46:01.687960882 +0000 UTC m=+332.007938247"
Jul 18 00:46:06.978209 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.978163    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"]
Jul 18 00:46:06.978735 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.978715    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0f3da3-413e-4957-b762-81714396b8e4" containerName="util"
Jul 18 00:46:06.978797 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.978740    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0f3da3-413e-4957-b762-81714396b8e4" containerName="util"
Jul 18 00:46:06.978797 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.978768    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0f3da3-413e-4957-b762-81714396b8e4" containerName="extract"
Jul 18 00:46:06.978797 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.978777    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0f3da3-413e-4957-b762-81714396b8e4" containerName="extract"
Jul 18 00:46:06.978934 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.978797    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0f3da3-413e-4957-b762-81714396b8e4" containerName="pull"
Jul 18 00:46:06.978934 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.978812    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0f3da3-413e-4957-b762-81714396b8e4" containerName="pull"
Jul 18 00:46:06.978934 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.978928    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0f3da3-413e-4957-b762-81714396b8e4" containerName="extract"
Jul 18 00:46:06.982483 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.982460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"
Jul 18 00:46:06.985025 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.984999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 18 00:46:06.986090 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.986037    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 18 00:46:06.986318 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.986127    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gwjhk\""
Jul 18 00:46:06.988049 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:06.988026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"]
Jul 18 00:46:07.032761 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.032726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8th\" (UniqueName: \"kubernetes.io/projected/3e7136b8-a288-4ba9-a2c3-17457bcf6e5b-kube-api-access-vk8th\") pod \"openshift-lws-operator-fd8ccff4c-m2mdr\" (UID: \"3e7136b8-a288-4ba9-a2c3-17457bcf6e5b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"
Jul 18 00:46:07.032941 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.032791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7136b8-a288-4ba9-a2c3-17457bcf6e5b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m2mdr\" (UID: \"3e7136b8-a288-4ba9-a2c3-17457bcf6e5b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"
Jul 18 00:46:07.134106 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.134045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7136b8-a288-4ba9-a2c3-17457bcf6e5b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m2mdr\" (UID: \"3e7136b8-a288-4ba9-a2c3-17457bcf6e5b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"
Jul 18 00:46:07.134269 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.134194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8th\" (UniqueName: \"kubernetes.io/projected/3e7136b8-a288-4ba9-a2c3-17457bcf6e5b-kube-api-access-vk8th\") pod \"openshift-lws-operator-fd8ccff4c-m2mdr\" (UID: \"3e7136b8-a288-4ba9-a2c3-17457bcf6e5b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"
Jul 18 00:46:07.134500 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.134475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7136b8-a288-4ba9-a2c3-17457bcf6e5b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m2mdr\" (UID: \"3e7136b8-a288-4ba9-a2c3-17457bcf6e5b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"
Jul 18 00:46:07.143020 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.142990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8th\" (UniqueName: \"kubernetes.io/projected/3e7136b8-a288-4ba9-a2c3-17457bcf6e5b-kube-api-access-vk8th\") pod \"openshift-lws-operator-fd8ccff4c-m2mdr\" (UID: \"3e7136b8-a288-4ba9-a2c3-17457bcf6e5b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"
Jul 18 00:46:07.291768 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.291677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"
Jul 18 00:46:07.425009 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.424964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr"]
Jul 18 00:46:07.428078 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:46:07.428034    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e7136b8_a288_4ba9_a2c3_17457bcf6e5b.slice/crio-29fd09f2effc13300bb4b6b88dc8ae8cb76f1a22c77f3b0c3dc815aae0d7c1c5 WatchSource:0}: Error finding container 29fd09f2effc13300bb4b6b88dc8ae8cb76f1a22c77f3b0c3dc815aae0d7c1c5: Status 404 returned error can't find the container with id 29fd09f2effc13300bb4b6b88dc8ae8cb76f1a22c77f3b0c3dc815aae0d7c1c5
Jul 18 00:46:07.691780 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:07.691686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr" event={"ID":"3e7136b8-a288-4ba9-a2c3-17457bcf6e5b","Type":"ContainerStarted","Data":"29fd09f2effc13300bb4b6b88dc8ae8cb76f1a22c77f3b0c3dc815aae0d7c1c5"}
Jul 18 00:46:09.701788 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:09.701748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr" event={"ID":"3e7136b8-a288-4ba9-a2c3-17457bcf6e5b","Type":"ContainerStarted","Data":"46d133d3e1b67d2c9f70c794f4bbddc5a2397f40949fbf27e23dd2efd91df74d"}
Jul 18 00:46:09.717378 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:09.717310    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m2mdr" podStartSLOduration=2.021803014 podStartE2EDuration="3.717289934s" totalImagesPullingTime="1.69548692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:09.716503303 +0000 UTC m=+340.036480686" watchObservedRunningTime="2026-07-18 00:46:09.717289934 +0000 UTC m=+340.037267316"
Jul 18 00:46:12.639981 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.639940    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"]
Jul 18 00:46:12.643569 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.643543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.645896 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.645870    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 18 00:46:12.646016 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.645930    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 18 00:46:12.646822 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.646806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5w4gn\""
Jul 18 00:46:12.650696 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.650673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"]
Jul 18 00:46:12.673333 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.673297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.673485 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.673340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.673485 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.673406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6bc\" (UniqueName: \"kubernetes.io/projected/f0f33af8-dcce-445e-9604-5d6cc90d4152-kube-api-access-vb6bc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.773979 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.773939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.774186 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.774015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6bc\" (UniqueName: \"kubernetes.io/projected/f0f33af8-dcce-445e-9604-5d6cc90d4152-kube-api-access-vb6bc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.774186 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.774096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.774463 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.774441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.774502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.774453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.782232 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.782203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6bc\" (UniqueName: \"kubernetes.io/projected/f0f33af8-dcce-445e-9604-5d6cc90d4152-kube-api-access-vb6bc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:12.953761 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:12.953720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:13.085574 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:13.085546    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"]
Jul 18 00:46:13.088012 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:46:13.087979    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f33af8_dcce_445e_9604_5d6cc90d4152.slice/crio-f56392a2befd1eddd138db56d1daa1b5ad9d3bc4cda6187d397ece08353fd9e6 WatchSource:0}: Error finding container f56392a2befd1eddd138db56d1daa1b5ad9d3bc4cda6187d397ece08353fd9e6: Status 404 returned error can't find the container with id f56392a2befd1eddd138db56d1daa1b5ad9d3bc4cda6187d397ece08353fd9e6
Jul 18 00:46:13.719243 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:13.719206    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerID="d1662764af5354e8496f1c308eedf5eeaa66ee6ea9b7741245425241b00debcb" exitCode=0
Jul 18 00:46:13.719625 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:13.719296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6" event={"ID":"f0f33af8-dcce-445e-9604-5d6cc90d4152","Type":"ContainerDied","Data":"d1662764af5354e8496f1c308eedf5eeaa66ee6ea9b7741245425241b00debcb"}
Jul 18 00:46:13.719625 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:13.719350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6" event={"ID":"f0f33af8-dcce-445e-9604-5d6cc90d4152","Type":"ContainerStarted","Data":"f56392a2befd1eddd138db56d1daa1b5ad9d3bc4cda6187d397ece08353fd9e6"}
Jul 18 00:46:14.724838 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:14.724797    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerID="41487e0a9cff96951c6890c6970613d69c0bf875fdc1f118db009fcf9204a817" exitCode=0
Jul 18 00:46:14.725274 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:14.724858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6" event={"ID":"f0f33af8-dcce-445e-9604-5d6cc90d4152","Type":"ContainerDied","Data":"41487e0a9cff96951c6890c6970613d69c0bf875fdc1f118db009fcf9204a817"}
Jul 18 00:46:15.730681 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:15.730643    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerID="b1e1fb624228ed8b1c7e28efbab27b18bd673bcb001c2a1a68c5189366abf1a7" exitCode=0
Jul 18 00:46:15.731070 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:15.730729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6" event={"ID":"f0f33af8-dcce-445e-9604-5d6cc90d4152","Type":"ContainerDied","Data":"b1e1fb624228ed8b1c7e28efbab27b18bd673bcb001c2a1a68c5189366abf1a7"}
Jul 18 00:46:16.871537 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:16.871509    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:16.908528 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:16.908495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-util\") pod \"f0f33af8-dcce-445e-9604-5d6cc90d4152\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") "
Jul 18 00:46:16.908692 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:16.908540    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb6bc\" (UniqueName: \"kubernetes.io/projected/f0f33af8-dcce-445e-9604-5d6cc90d4152-kube-api-access-vb6bc\") pod \"f0f33af8-dcce-445e-9604-5d6cc90d4152\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") "
Jul 18 00:46:16.908692 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:16.908576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-bundle\") pod \"f0f33af8-dcce-445e-9604-5d6cc90d4152\" (UID: \"f0f33af8-dcce-445e-9604-5d6cc90d4152\") "
Jul 18 00:46:16.909367 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:16.909339    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-bundle" (OuterVolumeSpecName: "bundle") pod "f0f33af8-dcce-445e-9604-5d6cc90d4152" (UID: "f0f33af8-dcce-445e-9604-5d6cc90d4152"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:46:16.910690 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:16.910656    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f33af8-dcce-445e-9604-5d6cc90d4152-kube-api-access-vb6bc" (OuterVolumeSpecName: "kube-api-access-vb6bc") pod "f0f33af8-dcce-445e-9604-5d6cc90d4152" (UID: "f0f33af8-dcce-445e-9604-5d6cc90d4152"). InnerVolumeSpecName "kube-api-access-vb6bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:46:16.914342 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:16.914248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-util" (OuterVolumeSpecName: "util") pod "f0f33af8-dcce-445e-9604-5d6cc90d4152" (UID: "f0f33af8-dcce-445e-9604-5d6cc90d4152"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:46:17.009523 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:17.009441    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-util\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:46:17.009523 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:17.009473    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6bc\" (UniqueName: \"kubernetes.io/projected/f0f33af8-dcce-445e-9604-5d6cc90d4152-kube-api-access-vb6bc\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:46:17.009523 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:17.009484    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f33af8-dcce-445e-9604-5d6cc90d4152-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:46:17.740802 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:17.740763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6" event={"ID":"f0f33af8-dcce-445e-9604-5d6cc90d4152","Type":"ContainerDied","Data":"f56392a2befd1eddd138db56d1daa1b5ad9d3bc4cda6187d397ece08353fd9e6"}
Jul 18 00:46:17.740976 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:17.740824    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f56392a2befd1eddd138db56d1daa1b5ad9d3bc4cda6187d397ece08353fd9e6"
Jul 18 00:46:17.740976 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:17.740786    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxbqf6"
Jul 18 00:46:29.305428 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.305349    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"]
Jul 18 00:46:29.305796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.305618    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerName="util"
Jul 18 00:46:29.305796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.305629    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerName="util"
Jul 18 00:46:29.305796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.305637    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerName="pull"
Jul 18 00:46:29.305796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.305642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerName="pull"
Jul 18 00:46:29.305796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.305648    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerName="extract"
Jul 18 00:46:29.305796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.305653    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerName="extract"
Jul 18 00:46:29.305796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.305706    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f33af8-dcce-445e-9604-5d6cc90d4152" containerName="extract"
Jul 18 00:46:29.309934 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.309916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.314773 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.314744    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 18 00:46:29.316681 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.316661    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 18 00:46:29.316965 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.316948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-46br2\""
Jul 18 00:46:29.346508 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.346462    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 18 00:46:29.346508 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.346493    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 18 00:46:29.357152 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.357117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"]
Jul 18 00:46:29.398487 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.398448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28a1cf60-87ae-450c-b575-0c7bb88cf938-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.398487 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.398490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28a1cf60-87ae-450c-b575-0c7bb88cf938-manifests\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.398723 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.398573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28a1cf60-87ae-450c-b575-0c7bb88cf938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.398723 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.398634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28a1cf60-87ae-450c-b575-0c7bb88cf938-tmp\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.398723 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.398688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rn74\" (UniqueName: \"kubernetes.io/projected/28a1cf60-87ae-450c-b575-0c7bb88cf938-kube-api-access-4rn74\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.500152 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.500115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rn74\" (UniqueName: \"kubernetes.io/projected/28a1cf60-87ae-450c-b575-0c7bb88cf938-kube-api-access-4rn74\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.500345 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.500163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28a1cf60-87ae-450c-b575-0c7bb88cf938-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.500345 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.500182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28a1cf60-87ae-450c-b575-0c7bb88cf938-manifests\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.500345 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.500235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28a1cf60-87ae-450c-b575-0c7bb88cf938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.500345 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.500265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28a1cf60-87ae-450c-b575-0c7bb88cf938-tmp\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.500637 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.500611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/28a1cf60-87ae-450c-b575-0c7bb88cf938-manifests\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.502621 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.502598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28a1cf60-87ae-450c-b575-0c7bb88cf938-tmp\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.502861 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.502841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28a1cf60-87ae-450c-b575-0c7bb88cf938-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.502961 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.502938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/28a1cf60-87ae-450c-b575-0c7bb88cf938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.510895 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.510871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rn74\" (UniqueName: \"kubernetes.io/projected/28a1cf60-87ae-450c-b575-0c7bb88cf938-kube-api-access-4rn74\") pod \"opendatahub-operator-controller-manager-85c6dd979f-nzm74\" (UID: \"28a1cf60-87ae-450c-b575-0c7bb88cf938\") " pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.619559 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.619465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:29.752874 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.752834    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"]
Jul 18 00:46:29.755813 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:46:29.755788    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a1cf60_87ae_450c_b575_0c7bb88cf938.slice/crio-1781717f714d3067c9ff5ca82efdfb59391d2f99f5958d18f150cc847a5e74cc WatchSource:0}: Error finding container 1781717f714d3067c9ff5ca82efdfb59391d2f99f5958d18f150cc847a5e74cc: Status 404 returned error can't find the container with id 1781717f714d3067c9ff5ca82efdfb59391d2f99f5958d18f150cc847a5e74cc
Jul 18 00:46:29.795319 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:29.795278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74" event={"ID":"28a1cf60-87ae-450c-b575-0c7bb88cf938","Type":"ContainerStarted","Data":"1781717f714d3067c9ff5ca82efdfb59391d2f99f5958d18f150cc847a5e74cc"}
Jul 18 00:46:33.819909 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:33.819867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74" event={"ID":"28a1cf60-87ae-450c-b575-0c7bb88cf938","Type":"ContainerStarted","Data":"0662503a52ba90ccf3f61d07989e56bac33e795412b9029367a0d6a086878c5f"}
Jul 18 00:46:34.837949 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:34.837910    2563 generic.go:356] "Generic (PLEG): container finished" podID="28a1cf60-87ae-450c-b575-0c7bb88cf938" containerID="0662503a52ba90ccf3f61d07989e56bac33e795412b9029367a0d6a086878c5f" exitCode=0
Jul 18 00:46:34.838360 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:34.837981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74" event={"ID":"28a1cf60-87ae-450c-b575-0c7bb88cf938","Type":"ContainerDied","Data":"0662503a52ba90ccf3f61d07989e56bac33e795412b9029367a0d6a086878c5f"}
Jul 18 00:46:35.844106 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:35.844039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74" event={"ID":"28a1cf60-87ae-450c-b575-0c7bb88cf938","Type":"ContainerStarted","Data":"fc27facc8e81e69bf227f5887813ad9c20a329c15ea6ea4f65986ee3d310f361"}
Jul 18 00:46:35.844486 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:35.844148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:46.850790 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:46.850754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74"
Jul 18 00:46:46.874874 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:46.874816    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85c6dd979f-nzm74" podStartSLOduration=13.338691659 podStartE2EDuration="17.87479641s" totalImagesPullingTime="3.452705604s" totalInitContainerRuntime="1.083399147s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:35.869994091 +0000 UTC m=+366.189971479" watchObservedRunningTime="2026-07-18 00:46:46.87479641 +0000 UTC m=+377.194773794"
Jul 18 00:46:49.812485 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.812449    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"]
Jul 18 00:46:49.821203 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.821180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:49.823818 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.823799    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 18 00:46:49.823926 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.823800    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 18 00:46:49.824092 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.824048    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 18 00:46:49.824092 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.824087    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 18 00:46:49.824852 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.824835    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8fvc4\""
Jul 18 00:46:49.829828 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.829805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"]
Jul 18 00:46:49.969515 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.969478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe113b3-f563-4819-85db-43159544559b-tmp\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:49.969515 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.969525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe113b3-f563-4819-85db-43159544559b-tls-certs\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:49.969757 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:49.969561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797bx\" (UniqueName: \"kubernetes.io/projected/fbe113b3-f563-4819-85db-43159544559b-kube-api-access-797bx\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:50.070078 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.069967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797bx\" (UniqueName: \"kubernetes.io/projected/fbe113b3-f563-4819-85db-43159544559b-kube-api-access-797bx\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:50.070231 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.070088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe113b3-f563-4819-85db-43159544559b-tmp\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:50.070231 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.070127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe113b3-f563-4819-85db-43159544559b-tls-certs\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:50.072458 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.072425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe113b3-f563-4819-85db-43159544559b-tmp\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:50.072644 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.072626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe113b3-f563-4819-85db-43159544559b-tls-certs\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:50.079506 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.079482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-797bx\" (UniqueName: \"kubernetes.io/projected/fbe113b3-f563-4819-85db-43159544559b-kube-api-access-797bx\") pod \"kube-auth-proxy-5f655d47ff-phrzs\" (UID: \"fbe113b3-f563-4819-85db-43159544559b\") " pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:50.130872 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.130834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"
Jul 18 00:46:50.204171 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.204134    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2r5ds"]
Jul 18 00:46:50.208038 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.208013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:50.211261 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.211208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-47zf2\""
Jul 18 00:46:50.211572 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.211554    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 18 00:46:50.231036 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.230535    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2r5ds"]
Jul 18 00:46:50.272049 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.272014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b94f72-c9ff-417d-a757-17c2f33d642d-tls-certs\") pod \"model-serving-api-54d89bd5fc-2r5ds\" (UID: \"50b94f72-c9ff-417d-a757-17c2f33d642d\") " pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:50.272049 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.272138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xws8\" (UniqueName: \"kubernetes.io/projected/50b94f72-c9ff-417d-a757-17c2f33d642d-kube-api-access-2xws8\") pod \"model-serving-api-54d89bd5fc-2r5ds\" (UID: \"50b94f72-c9ff-417d-a757-17c2f33d642d\") " pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:50.286029 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.286003    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs"]
Jul 18 00:46:50.289349 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:46:50.289319    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe113b3_f563_4819_85db_43159544559b.slice/crio-4c6c5b260ec10db575c6dc54959fff596ebe3facdf3a27587e9e2098e887db82 WatchSource:0}: Error finding container 4c6c5b260ec10db575c6dc54959fff596ebe3facdf3a27587e9e2098e887db82: Status 404 returned error can't find the container with id 4c6c5b260ec10db575c6dc54959fff596ebe3facdf3a27587e9e2098e887db82
Jul 18 00:46:50.372890 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.372781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b94f72-c9ff-417d-a757-17c2f33d642d-tls-certs\") pod \"model-serving-api-54d89bd5fc-2r5ds\" (UID: \"50b94f72-c9ff-417d-a757-17c2f33d642d\") " pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:50.372890 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.372837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xws8\" (UniqueName: \"kubernetes.io/projected/50b94f72-c9ff-417d-a757-17c2f33d642d-kube-api-access-2xws8\") pod \"model-serving-api-54d89bd5fc-2r5ds\" (UID: \"50b94f72-c9ff-417d-a757-17c2f33d642d\") " pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:50.373181 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:46:50.372946    2563 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 18 00:46:50.373181 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:46:50.373029    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b94f72-c9ff-417d-a757-17c2f33d642d-tls-certs podName:50b94f72-c9ff-417d-a757-17c2f33d642d nodeName:}" failed. No retries permitted until 2026-07-18 00:46:50.873005791 +0000 UTC m=+381.192983170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/50b94f72-c9ff-417d-a757-17c2f33d642d-tls-certs") pod "model-serving-api-54d89bd5fc-2r5ds" (UID: "50b94f72-c9ff-417d-a757-17c2f33d642d") : secret "model-serving-api-tls" not found
Jul 18 00:46:50.382497 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.382467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xws8\" (UniqueName: \"kubernetes.io/projected/50b94f72-c9ff-417d-a757-17c2f33d642d-kube-api-access-2xws8\") pod \"model-serving-api-54d89bd5fc-2r5ds\" (UID: \"50b94f72-c9ff-417d-a757-17c2f33d642d\") " pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:50.877841 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.877798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b94f72-c9ff-417d-a757-17c2f33d642d-tls-certs\") pod \"model-serving-api-54d89bd5fc-2r5ds\" (UID: \"50b94f72-c9ff-417d-a757-17c2f33d642d\") " pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:50.880199 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.880180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b94f72-c9ff-417d-a757-17c2f33d642d-tls-certs\") pod \"model-serving-api-54d89bd5fc-2r5ds\" (UID: \"50b94f72-c9ff-417d-a757-17c2f33d642d\") " pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:50.911074 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.911014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs" event={"ID":"fbe113b3-f563-4819-85db-43159544559b","Type":"ContainerStarted","Data":"4c6c5b260ec10db575c6dc54959fff596ebe3facdf3a27587e9e2098e887db82"}
Jul 18 00:46:50.975661 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.975622    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"]
Jul 18 00:46:50.979265 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.979248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:50.982036 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.982008    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 18 00:46:50.982227 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.982212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 18 00:46:50.982384 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.982367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6rkdc\""
Jul 18 00:46:50.982597 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.982578    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 18 00:46:50.982677 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:50.982633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 00:46:51.021091 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.021048    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"]
Jul 18 00:46:51.079599 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.079562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbae8e51-7d45-414a-962e-544ed62d49e1-local-certs\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.079599 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.079605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.079872 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.079625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.079872 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.079645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrpwg\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-kube-api-access-zrpwg\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.079872 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.079688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.079872 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.079725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-cacerts\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.079872 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.079758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-token\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.122811 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.122771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:51.182392 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.182155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbae8e51-7d45-414a-962e-544ed62d49e1-local-certs\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.182392 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.182202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.182392 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.182234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.182392 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.182260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrpwg\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-kube-api-access-zrpwg\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.182392 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.182290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.182392 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.182322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-cacerts\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.182392 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.182355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-token\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.183967 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.183939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.185925 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.185897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-cacerts\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.186239 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.186220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.187233 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.187193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.190531 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.190487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbae8e51-7d45-414a-962e-544ed62d49e1-local-certs\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.191725 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.191693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-token\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.194976 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.194941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrpwg\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-kube-api-access-zrpwg\") pod \"istiod-openshift-gateway-7574459656-mb44s\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.278735 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.278653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2r5ds"]
Jul 18 00:46:51.281214 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:46:51.281184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b94f72_c9ff_417d_a757_17c2f33d642d.slice/crio-eb5323cba164b6471d43a6b96c55952b55b6b41fc3126f539d3138fc776a6bd7 WatchSource:0}: Error finding container eb5323cba164b6471d43a6b96c55952b55b6b41fc3126f539d3138fc776a6bd7: Status 404 returned error can't find the container with id eb5323cba164b6471d43a6b96c55952b55b6b41fc3126f539d3138fc776a6bd7
Jul 18 00:46:51.288996 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.288943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:51.437399 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.437370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"]
Jul 18 00:46:51.440621 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:46:51.440593    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbae8e51_7d45_414a_962e_544ed62d49e1.slice/crio-5777ddb31b732efca712fc020cfb62dbd5d710f4dfb2a08b5b957c0b2ede9c68 WatchSource:0}: Error finding container 5777ddb31b732efca712fc020cfb62dbd5d710f4dfb2a08b5b957c0b2ede9c68: Status 404 returned error can't find the container with id 5777ddb31b732efca712fc020cfb62dbd5d710f4dfb2a08b5b957c0b2ede9c68
Jul 18 00:46:51.920796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.920705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds" event={"ID":"50b94f72-c9ff-417d-a757-17c2f33d642d","Type":"ContainerStarted","Data":"eb5323cba164b6471d43a6b96c55952b55b6b41fc3126f539d3138fc776a6bd7"}
Jul 18 00:46:51.922695 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:51.922651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s" event={"ID":"cbae8e51-7d45-414a-962e-544ed62d49e1","Type":"ContainerStarted","Data":"5777ddb31b732efca712fc020cfb62dbd5d710f4dfb2a08b5b957c0b2ede9c68"}
Jul 18 00:46:54.643804 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.643773    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 18 00:46:54.644135 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.643851    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 18 00:46:54.939862 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.939757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs" event={"ID":"fbe113b3-f563-4819-85db-43159544559b","Type":"ContainerStarted","Data":"678ea377d63186721a7303366b8e3026589cb592296b42fed6ae3139069ab06a"}
Jul 18 00:46:54.941394 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.941365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds" event={"ID":"50b94f72-c9ff-417d-a757-17c2f33d642d","Type":"ContainerStarted","Data":"02e5c27c8d2f643b599ef7e088d6e34806220378f2088bb03fc0276813812630"}
Jul 18 00:46:54.941521 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.941426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:46:54.943138 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.943115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s" event={"ID":"cbae8e51-7d45-414a-962e-544ed62d49e1","Type":"ContainerStarted","Data":"c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f"}
Jul 18 00:46:54.943271 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.943230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:46:54.955691 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.955641    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f655d47ff-phrzs" podStartSLOduration=1.613141187 podStartE2EDuration="5.955623542s" totalImagesPullingTime="4.342482355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:54.954660958 +0000 UTC m=+385.274638340" watchObservedRunningTime="2026-07-18 00:46:54.955623542 +0000 UTC m=+385.275600926"
Jul 18 00:46:54.976132 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.976038    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s" podStartSLOduration=1.775286455 podStartE2EDuration="4.976015489s" totalImagesPullingTime="3.200729034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:54.973818459 +0000 UTC m=+385.293795843" watchObservedRunningTime="2026-07-18 00:46:54.976015489 +0000 UTC m=+385.295992873"
Jul 18 00:46:54.988923 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:54.988874    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds" podStartSLOduration=1.638393035 podStartE2EDuration="4.988856661s" totalImagesPullingTime="3.350463626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:54.987816293 +0000 UTC m=+385.307793703" watchObservedRunningTime="2026-07-18 00:46:54.988856661 +0000 UTC m=+385.308834044"
Jul 18 00:46:55.949625 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:55.949593    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-mb44s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 18 00:46:55.949997 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:55.949644    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s" podUID="cbae8e51-7d45-414a-962e-544ed62d49e1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 00:46:58.949153 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:46:58.949113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:47:01.253812 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.253775    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"]
Jul 18 00:47:01.257499 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.257475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:01.264041 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.264012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-s5mpj\""
Jul 18 00:47:01.265202 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.264867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 18 00:47:01.270890 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.270866    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"]
Jul 18 00:47:01.273857 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.273841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:01.278463 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.278409    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 18 00:47:01.278463 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.278439    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zt7p6\""
Jul 18 00:47:01.295517 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.295487    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"]
Jul 18 00:47:01.332041 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.332009    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"]
Jul 18 00:47:01.371933 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.371895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dqs\" (UniqueName: \"kubernetes.io/projected/c8b41baa-e1f0-4ccc-9fa9-00e46c4af052-kube-api-access-b6dqs\") pod \"kserve-controller-manager-d6f4b65b8-jgrhr\" (UID: \"c8b41baa-e1f0-4ccc-9fa9-00e46c4af052\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:01.372180 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.371955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fmcn7\" (UID: \"444186e3-c146-4024-b7fb-e99ecbe93a58\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:01.372180 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.372005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8b41baa-e1f0-4ccc-9fa9-00e46c4af052-cert\") pod \"kserve-controller-manager-d6f4b65b8-jgrhr\" (UID: \"c8b41baa-e1f0-4ccc-9fa9-00e46c4af052\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:01.372180 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.372122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4q69\" (UniqueName: \"kubernetes.io/projected/444186e3-c146-4024-b7fb-e99ecbe93a58-kube-api-access-v4q69\") pod \"llmisvc-controller-manager-777c6bd7d5-fmcn7\" (UID: \"444186e3-c146-4024-b7fb-e99ecbe93a58\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:01.473121 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.473087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4q69\" (UniqueName: \"kubernetes.io/projected/444186e3-c146-4024-b7fb-e99ecbe93a58-kube-api-access-v4q69\") pod \"llmisvc-controller-manager-777c6bd7d5-fmcn7\" (UID: \"444186e3-c146-4024-b7fb-e99ecbe93a58\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:01.473290 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.473152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dqs\" (UniqueName: \"kubernetes.io/projected/c8b41baa-e1f0-4ccc-9fa9-00e46c4af052-kube-api-access-b6dqs\") pod \"kserve-controller-manager-d6f4b65b8-jgrhr\" (UID: \"c8b41baa-e1f0-4ccc-9fa9-00e46c4af052\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:01.473290 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.473197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fmcn7\" (UID: \"444186e3-c146-4024-b7fb-e99ecbe93a58\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:01.473290 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.473248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8b41baa-e1f0-4ccc-9fa9-00e46c4af052-cert\") pod \"kserve-controller-manager-d6f4b65b8-jgrhr\" (UID: \"c8b41baa-e1f0-4ccc-9fa9-00e46c4af052\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:01.473441 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:47:01.473395    2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 18 00:47:01.473491 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:47:01.473468    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert podName:444186e3-c146-4024-b7fb-e99ecbe93a58 nodeName:}" failed. No retries permitted until 2026-07-18 00:47:01.973445437 +0000 UTC m=+392.293422812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert") pod "llmisvc-controller-manager-777c6bd7d5-fmcn7" (UID: "444186e3-c146-4024-b7fb-e99ecbe93a58") : secret "llmisvc-webhook-server-cert" not found
Jul 18 00:47:01.475829 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.475799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8b41baa-e1f0-4ccc-9fa9-00e46c4af052-cert\") pod \"kserve-controller-manager-d6f4b65b8-jgrhr\" (UID: \"c8b41baa-e1f0-4ccc-9fa9-00e46c4af052\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:01.487641 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.487604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dqs\" (UniqueName: \"kubernetes.io/projected/c8b41baa-e1f0-4ccc-9fa9-00e46c4af052-kube-api-access-b6dqs\") pod \"kserve-controller-manager-d6f4b65b8-jgrhr\" (UID: \"c8b41baa-e1f0-4ccc-9fa9-00e46c4af052\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:01.487641 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.487635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4q69\" (UniqueName: \"kubernetes.io/projected/444186e3-c146-4024-b7fb-e99ecbe93a58-kube-api-access-v4q69\") pod \"llmisvc-controller-manager-777c6bd7d5-fmcn7\" (UID: \"444186e3-c146-4024-b7fb-e99ecbe93a58\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:01.568582 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.568475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:01.730806 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.730761    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"]
Jul 18 00:47:01.732641 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:47:01.732616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b41baa_e1f0_4ccc_9fa9_00e46c4af052.slice/crio-92de49b165f1437971fcd63b0f399b9798d2066b54ef00e2bf01dc7e913bda6d WatchSource:0}: Error finding container 92de49b165f1437971fcd63b0f399b9798d2066b54ef00e2bf01dc7e913bda6d: Status 404 returned error can't find the container with id 92de49b165f1437971fcd63b0f399b9798d2066b54ef00e2bf01dc7e913bda6d
Jul 18 00:47:01.975130 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.975092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr" event={"ID":"c8b41baa-e1f0-4ccc-9fa9-00e46c4af052","Type":"ContainerStarted","Data":"92de49b165f1437971fcd63b0f399b9798d2066b54ef00e2bf01dc7e913bda6d"}
Jul 18 00:47:01.977517 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:01.977493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fmcn7\" (UID: \"444186e3-c146-4024-b7fb-e99ecbe93a58\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:01.977650 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:47:01.977633    2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 18 00:47:01.977701 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:47:01.977692    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert podName:444186e3-c146-4024-b7fb-e99ecbe93a58 nodeName:}" failed. No retries permitted until 2026-07-18 00:47:02.977677695 +0000 UTC m=+393.297655057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert") pod "llmisvc-controller-manager-777c6bd7d5-fmcn7" (UID: "444186e3-c146-4024-b7fb-e99ecbe93a58") : secret "llmisvc-webhook-server-cert" not found
Jul 18 00:47:02.987424 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:02.987389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fmcn7\" (UID: \"444186e3-c146-4024-b7fb-e99ecbe93a58\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:02.990375 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:02.990346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444186e3-c146-4024-b7fb-e99ecbe93a58-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fmcn7\" (UID: \"444186e3-c146-4024-b7fb-e99ecbe93a58\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:03.083571 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:03.083532    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:03.288239 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:03.288202    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"]
Jul 18 00:47:03.288392 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:47:03.288366    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod444186e3_c146_4024_b7fb_e99ecbe93a58.slice/crio-c30dab24227168b1b5af0ee81fdae782b7f62ddb98f73ee34a9e2392c9b2222f WatchSource:0}: Error finding container c30dab24227168b1b5af0ee81fdae782b7f62ddb98f73ee34a9e2392c9b2222f: Status 404 returned error can't find the container with id c30dab24227168b1b5af0ee81fdae782b7f62ddb98f73ee34a9e2392c9b2222f
Jul 18 00:47:03.988614 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:03.988576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7" event={"ID":"444186e3-c146-4024-b7fb-e99ecbe93a58","Type":"ContainerStarted","Data":"c30dab24227168b1b5af0ee81fdae782b7f62ddb98f73ee34a9e2392c9b2222f"}
Jul 18 00:47:04.110862 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.110820    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"]
Jul 18 00:47:04.111051 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.111021    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s" podUID="cbae8e51-7d45-414a-962e-544ed62d49e1" containerName="discovery" containerID="cri-o://c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f" gracePeriod=30
Jul 18 00:47:04.361018 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.360992    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:47:04.501876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.501801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-dns-cert\") pod \"cbae8e51-7d45-414a-962e-544ed62d49e1\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") "
Jul 18 00:47:04.501876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.501841    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-ca-configmap\") pod \"cbae8e51-7d45-414a-962e-544ed62d49e1\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") "
Jul 18 00:47:04.501876 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.501872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-cacerts\") pod \"cbae8e51-7d45-414a-962e-544ed62d49e1\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") "
Jul 18 00:47:04.502172 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.502027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-kubeconfig\") pod \"cbae8e51-7d45-414a-962e-544ed62d49e1\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") "
Jul 18 00:47:04.502172 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.502115    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-token\") pod \"cbae8e51-7d45-414a-962e-544ed62d49e1\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") "
Jul 18 00:47:04.502172 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.502152    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbae8e51-7d45-414a-962e-544ed62d49e1-local-certs\") pod \"cbae8e51-7d45-414a-962e-544ed62d49e1\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") "
Jul 18 00:47:04.502310 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.502176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrpwg\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-kube-api-access-zrpwg\") pod \"cbae8e51-7d45-414a-962e-544ed62d49e1\" (UID: \"cbae8e51-7d45-414a-962e-544ed62d49e1\") "
Jul 18 00:47:04.502380 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.502355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cbae8e51-7d45-414a-962e-544ed62d49e1" (UID: "cbae8e51-7d45-414a-962e-544ed62d49e1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:47:04.502495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.502464    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-ca-configmap\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:47:04.504917 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.504860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbae8e51-7d45-414a-962e-544ed62d49e1-local-certs" (OuterVolumeSpecName: "local-certs") pod "cbae8e51-7d45-414a-962e-544ed62d49e1" (UID: "cbae8e51-7d45-414a-962e-544ed62d49e1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:47:04.504917 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.504896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cbae8e51-7d45-414a-962e-544ed62d49e1" (UID: "cbae8e51-7d45-414a-962e-544ed62d49e1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:47:04.505138 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.504929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-kube-api-access-zrpwg" (OuterVolumeSpecName: "kube-api-access-zrpwg") pod "cbae8e51-7d45-414a-962e-544ed62d49e1" (UID: "cbae8e51-7d45-414a-962e-544ed62d49e1"). InnerVolumeSpecName "kube-api-access-zrpwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:47:04.505138 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.504953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-cacerts" (OuterVolumeSpecName: "cacerts") pod "cbae8e51-7d45-414a-962e-544ed62d49e1" (UID: "cbae8e51-7d45-414a-962e-544ed62d49e1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:47:04.505138 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.505018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbae8e51-7d45-414a-962e-544ed62d49e1" (UID: "cbae8e51-7d45-414a-962e-544ed62d49e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:47:04.505138 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.505102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cbae8e51-7d45-414a-962e-544ed62d49e1" (UID: "cbae8e51-7d45-414a-962e-544ed62d49e1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:47:04.605081 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.603849    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-csr-dns-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:47:04.605081 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.603899    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-cacerts\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:47:04.605081 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.603916    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-kubeconfig\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:47:04.605081 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.603930    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-istio-token\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:47:04.605081 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.603945    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbae8e51-7d45-414a-962e-544ed62d49e1-local-certs\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:47:04.605081 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.603962    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrpwg\" (UniqueName: \"kubernetes.io/projected/cbae8e51-7d45-414a-962e-544ed62d49e1-kube-api-access-zrpwg\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:47:04.997682 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.997647    2563 generic.go:356] "Generic (PLEG): container finished" podID="cbae8e51-7d45-414a-962e-544ed62d49e1" containerID="c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f" exitCode=0
Jul 18 00:47:04.998116 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.997712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"
Jul 18 00:47:04.998116 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.997736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s" event={"ID":"cbae8e51-7d45-414a-962e-544ed62d49e1","Type":"ContainerDied","Data":"c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f"}
Jul 18 00:47:04.998116 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.997829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mb44s" event={"ID":"cbae8e51-7d45-414a-962e-544ed62d49e1","Type":"ContainerDied","Data":"5777ddb31b732efca712fc020cfb62dbd5d710f4dfb2a08b5b957c0b2ede9c68"}
Jul 18 00:47:04.998116 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:04.997858    2563 scope.go:117] "RemoveContainer" containerID="c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f"
Jul 18 00:47:05.000415 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:05.000375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr" event={"ID":"c8b41baa-e1f0-4ccc-9fa9-00e46c4af052","Type":"ContainerStarted","Data":"ab2f15e79fec9e1d010e4708142ad6795e9a920264c54f4e87a71f4357f65fb1"}
Jul 18 00:47:05.000531 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:05.000522    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:05.039077 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:05.038878    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr" podStartSLOduration=1.5650429510000001 podStartE2EDuration="4.038855536s" totalImagesPullingTime="2.473812585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:47:05.036838813 +0000 UTC m=+395.356816231" watchObservedRunningTime="2026-07-18 00:47:05.038855536 +0000 UTC m=+395.358832921"
Jul 18 00:47:05.067890 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:05.067864    2563 scope.go:117] "RemoveContainer" containerID="c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f"
Jul 18 00:47:05.068280 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:47:05.068257    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f\": container with ID starting with c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f not found: ID does not exist" containerID="c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f"
Jul 18 00:47:05.068402 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:05.068293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f"} err="failed to get container status \"c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f\": rpc error: code = NotFound desc = could not find container \"c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f\": container with ID starting with c6d0f33b854761411a2988fdb10c659f697ef176a7f295bcca841873420e070f not found: ID does not exist"
Jul 18 00:47:05.078868 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:05.078831    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"]
Jul 18 00:47:05.089702 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:05.089657    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mb44s"]
Jul 18 00:47:05.952708 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:05.952677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2r5ds"
Jul 18 00:47:06.176080 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:06.176029    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbae8e51-7d45-414a-962e-544ed62d49e1" path="/var/lib/kubelet/pods/cbae8e51-7d45-414a-962e-544ed62d49e1/volumes"
Jul 18 00:47:07.013776 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:07.013736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7" event={"ID":"444186e3-c146-4024-b7fb-e99ecbe93a58","Type":"ContainerStarted","Data":"4987a2e96a78fac61928ba33c375cdd360a94d53b3021a5c7e2c055cc612eb4a"}
Jul 18 00:47:07.013941 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:07.013861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:07.031212 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:07.031162    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7" podStartSLOduration=2.715291177 podStartE2EDuration="6.031146855s" totalImagesPullingTime="3.315855678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:47:07.028946433 +0000 UTC m=+397.348923817" watchObservedRunningTime="2026-07-18 00:47:07.031146855 +0000 UTC m=+397.351124238"
Jul 18 00:47:36.012236 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:36.012203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jgrhr"
Jul 18 00:47:38.020485 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:38.020449    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fmcn7"
Jul 18 00:47:50.045817 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.045779    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6pcgd"]
Jul 18 00:47:50.046214 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.046105    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbae8e51-7d45-414a-962e-544ed62d49e1" containerName="discovery"
Jul 18 00:47:50.046214 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.046117    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbae8e51-7d45-414a-962e-544ed62d49e1" containerName="discovery"
Jul 18 00:47:50.046214 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.046173    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbae8e51-7d45-414a-962e-544ed62d49e1" containerName="discovery"
Jul 18 00:47:50.056488 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.056454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd"
Jul 18 00:47:50.059233 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.059208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 18 00:47:50.059413 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.059389    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 18 00:47:50.059490 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.059452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6pcgd"]
Jul 18 00:47:50.060276 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.060251    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7zfh9\""
Jul 18 00:47:50.154533 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.154489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2tb\" (UniqueName: \"kubernetes.io/projected/1320bd64-dd35-4dfe-9916-f5b41046d11e-kube-api-access-7n2tb\") pod \"kuadrant-operator-catalog-6pcgd\" (UID: \"1320bd64-dd35-4dfe-9916-f5b41046d11e\") " pod="kuadrant-system/kuadrant-operator-catalog-6pcgd"
Jul 18 00:47:50.255426 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.255378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2tb\" (UniqueName: \"kubernetes.io/projected/1320bd64-dd35-4dfe-9916-f5b41046d11e-kube-api-access-7n2tb\") pod \"kuadrant-operator-catalog-6pcgd\" (UID: \"1320bd64-dd35-4dfe-9916-f5b41046d11e\") " pod="kuadrant-system/kuadrant-operator-catalog-6pcgd"
Jul 18 00:47:50.263700 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.263665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2tb\" (UniqueName: \"kubernetes.io/projected/1320bd64-dd35-4dfe-9916-f5b41046d11e-kube-api-access-7n2tb\") pod \"kuadrant-operator-catalog-6pcgd\" (UID: \"1320bd64-dd35-4dfe-9916-f5b41046d11e\") " pod="kuadrant-system/kuadrant-operator-catalog-6pcgd"
Jul 18 00:47:50.376010 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.375913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd"
Jul 18 00:47:50.406725 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.406690    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6pcgd"]
Jul 18 00:47:50.511458 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.511432    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6pcgd"]
Jul 18 00:47:50.514025 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:47:50.513994    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1320bd64_dd35_4dfe_9916_f5b41046d11e.slice/crio-52f6f96e057eaae693457fc28ed3077f2a1eabe06d37083082f5770cd17d0034 WatchSource:0}: Error finding container 52f6f96e057eaae693457fc28ed3077f2a1eabe06d37083082f5770cd17d0034: Status 404 returned error can't find the container with id 52f6f96e057eaae693457fc28ed3077f2a1eabe06d37083082f5770cd17d0034
Jul 18 00:47:50.609659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.609622    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mgl5"]
Jul 18 00:47:50.613810 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.613791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:47:50.619408 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.619376    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mgl5"]
Jul 18 00:47:50.759483 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.759443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxt9b\" (UniqueName: \"kubernetes.io/projected/d45e6e69-9468-4381-9e68-1a3c8f895e47-kube-api-access-sxt9b\") pod \"kuadrant-operator-catalog-7mgl5\" (UID: \"d45e6e69-9468-4381-9e68-1a3c8f895e47\") " pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:47:50.859948 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.859910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxt9b\" (UniqueName: \"kubernetes.io/projected/d45e6e69-9468-4381-9e68-1a3c8f895e47-kube-api-access-sxt9b\") pod \"kuadrant-operator-catalog-7mgl5\" (UID: \"d45e6e69-9468-4381-9e68-1a3c8f895e47\") " pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:47:50.867901 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.867864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxt9b\" (UniqueName: \"kubernetes.io/projected/d45e6e69-9468-4381-9e68-1a3c8f895e47-kube-api-access-sxt9b\") pod \"kuadrant-operator-catalog-7mgl5\" (UID: \"d45e6e69-9468-4381-9e68-1a3c8f895e47\") " pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:47:50.923779 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:50.923741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:47:51.068758 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:51.068729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mgl5"]
Jul 18 00:47:51.071225 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:47:51.071196    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45e6e69_9468_4381_9e68_1a3c8f895e47.slice/crio-e5c7304ef204de0ad57190daecf23159168caaa34f502025494fc0a934dbe2df WatchSource:0}: Error finding container e5c7304ef204de0ad57190daecf23159168caaa34f502025494fc0a934dbe2df: Status 404 returned error can't find the container with id e5c7304ef204de0ad57190daecf23159168caaa34f502025494fc0a934dbe2df
Jul 18 00:47:51.210486 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:51.210436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5" event={"ID":"d45e6e69-9468-4381-9e68-1a3c8f895e47","Type":"ContainerStarted","Data":"e5c7304ef204de0ad57190daecf23159168caaa34f502025494fc0a934dbe2df"}
Jul 18 00:47:51.211801 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:51.211773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd" event={"ID":"1320bd64-dd35-4dfe-9916-f5b41046d11e","Type":"ContainerStarted","Data":"52f6f96e057eaae693457fc28ed3077f2a1eabe06d37083082f5770cd17d0034"}
Jul 18 00:47:53.223979 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.223937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5" event={"ID":"d45e6e69-9468-4381-9e68-1a3c8f895e47","Type":"ContainerStarted","Data":"fb3cce12b5691904d88389737bccd280090fab7d9acb9968cc7373d7bcf26e6d"}
Jul 18 00:47:53.225473 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.225445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd" event={"ID":"1320bd64-dd35-4dfe-9916-f5b41046d11e","Type":"ContainerStarted","Data":"38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8"}
Jul 18 00:47:53.225590 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.225484    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd" podUID="1320bd64-dd35-4dfe-9916-f5b41046d11e" containerName="registry-server" containerID="cri-o://38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8" gracePeriod=2
Jul 18 00:47:53.240695 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.240650    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5" podStartSLOduration=1.731657002 podStartE2EDuration="3.240633566s" totalImagesPullingTime="1.508976564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:47:53.238541642 +0000 UTC m=+443.558519036" watchObservedRunningTime="2026-07-18 00:47:53.240633566 +0000 UTC m=+443.560610949"
Jul 18 00:47:53.253840 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.253786    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd" podStartSLOduration=1.18822137 podStartE2EDuration="3.253768511s" totalImagesPullingTime="2.065547141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:47:53.251844525 +0000 UTC m=+443.571821920" watchObservedRunningTime="2026-07-18 00:47:53.253768511 +0000 UTC m=+443.573745895"
Jul 18 00:47:53.470781 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.470746    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd"
Jul 18 00:47:53.585550 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.585446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n2tb\" (UniqueName: \"kubernetes.io/projected/1320bd64-dd35-4dfe-9916-f5b41046d11e-kube-api-access-7n2tb\") pod \"1320bd64-dd35-4dfe-9916-f5b41046d11e\" (UID: \"1320bd64-dd35-4dfe-9916-f5b41046d11e\") "
Jul 18 00:47:53.587713 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.587683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1320bd64-dd35-4dfe-9916-f5b41046d11e-kube-api-access-7n2tb" (OuterVolumeSpecName: "kube-api-access-7n2tb") pod "1320bd64-dd35-4dfe-9916-f5b41046d11e" (UID: "1320bd64-dd35-4dfe-9916-f5b41046d11e"). InnerVolumeSpecName "kube-api-access-7n2tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:47:53.686287 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:53.686235    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n2tb\" (UniqueName: \"kubernetes.io/projected/1320bd64-dd35-4dfe-9916-f5b41046d11e-kube-api-access-7n2tb\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:47:54.230973 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.230935    2563 generic.go:356] "Generic (PLEG): container finished" podID="1320bd64-dd35-4dfe-9916-f5b41046d11e" containerID="38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8" exitCode=0
Jul 18 00:47:54.231461 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.230983    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd"
Jul 18 00:47:54.231461 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.231021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd" event={"ID":"1320bd64-dd35-4dfe-9916-f5b41046d11e","Type":"ContainerDied","Data":"38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8"}
Jul 18 00:47:54.231461 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.231100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6pcgd" event={"ID":"1320bd64-dd35-4dfe-9916-f5b41046d11e","Type":"ContainerDied","Data":"52f6f96e057eaae693457fc28ed3077f2a1eabe06d37083082f5770cd17d0034"}
Jul 18 00:47:54.231461 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.231123    2563 scope.go:117] "RemoveContainer" containerID="38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8"
Jul 18 00:47:54.240528 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.240510    2563 scope.go:117] "RemoveContainer" containerID="38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8"
Jul 18 00:47:54.240794 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:47:54.240774    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8\": container with ID starting with 38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8 not found: ID does not exist" containerID="38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8"
Jul 18 00:47:54.240838 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.240802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8"} err="failed to get container status \"38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8\": rpc error: code = NotFound desc = could not find container \"38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8\": container with ID starting with 38229aaad991d9f0c58323f75a5062f7facdef2ff8606a40c44a7355289aa6b8 not found: ID does not exist"
Jul 18 00:47:54.246439 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.246414    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6pcgd"]
Jul 18 00:47:54.249685 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:54.249663    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6pcgd"]
Jul 18 00:47:56.174840 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:47:56.174803    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1320bd64-dd35-4dfe-9916-f5b41046d11e" path="/var/lib/kubelet/pods/1320bd64-dd35-4dfe-9916-f5b41046d11e/volumes"
Jul 18 00:48:00.924751 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:00.924712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:48:00.925214 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:00.924786    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:48:00.947796 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:00.947765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:48:01.284949 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:01.284917    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7mgl5"
Jul 18 00:48:09.310148 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.310113    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"]
Jul 18 00:48:09.310587 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.310435    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1320bd64-dd35-4dfe-9916-f5b41046d11e" containerName="registry-server"
Jul 18 00:48:09.310587 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.310446    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1320bd64-dd35-4dfe-9916-f5b41046d11e" containerName="registry-server"
Jul 18 00:48:09.310587 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.310494    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1320bd64-dd35-4dfe-9916-f5b41046d11e" containerName="registry-server"
Jul 18 00:48:09.314698 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.314677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.317019 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.316995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-75dkv\""
Jul 18 00:48:09.320959 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.320933    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"]
Jul 18 00:48:09.408996 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.408948    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"]
Jul 18 00:48:09.412481 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.412453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.420071 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.419946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"]
Jul 18 00:48:09.424877 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.424849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg8fx\" (UniqueName: \"kubernetes.io/projected/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-kube-api-access-vg8fx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.424988 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.424907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.424988 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.424935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.526009 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.525568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pnbj\" (UniqueName: \"kubernetes.io/projected/7b0e438c-9503-437a-9114-aad368d08d6b-kube-api-access-6pnbj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.526351 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.526324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.526496 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.526481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.526650 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.526636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.526776 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.526761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.526873 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.526859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg8fx\" (UniqueName: \"kubernetes.io/projected/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-kube-api-access-vg8fx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.527836 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.527806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.528507 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.528484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.535064 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.535027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg8fx\" (UniqueName: \"kubernetes.io/projected/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-kube-api-access-vg8fx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.624485 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.624379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:09.627402 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.627369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pnbj\" (UniqueName: \"kubernetes.io/projected/7b0e438c-9503-437a-9114-aad368d08d6b-kube-api-access-6pnbj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.627536 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.627455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.627536 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.627497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.627960 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.627821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.627960 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.627851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.636786 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.636757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pnbj\" (UniqueName: \"kubernetes.io/projected/7b0e438c-9503-437a-9114-aad368d08d6b-kube-api-access-6pnbj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.707986 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.707951    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"]
Jul 18 00:48:09.713268 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.712941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.721007 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.720892    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"]
Jul 18 00:48:09.723311 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.723281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:09.771887 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.771589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"]
Jul 18 00:48:09.773787 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:09.773758    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9306e0e7_d6e3_42f6_98b2_e1f6316fc7ae.slice/crio-a9e050ee9ae6c72c27fa893006449072054f7b0d344c2ef30ab2fc489ef4d5dc WatchSource:0}: Error finding container a9e050ee9ae6c72c27fa893006449072054f7b0d344c2ef30ab2fc489ef4d5dc: Status 404 returned error can't find the container with id a9e050ee9ae6c72c27fa893006449072054f7b0d344c2ef30ab2fc489ef4d5dc
Jul 18 00:48:09.828813 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.828710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrtd\" (UniqueName: \"kubernetes.io/projected/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-kube-api-access-lwrtd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.829278 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.829225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.829509 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.829443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.882857 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.882827    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"]
Jul 18 00:48:09.885299 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:09.885268    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0e438c_9503_437a_9114_aad368d08d6b.slice/crio-758a54206c06f1e19401d39ee89cc04613d63fe9aa07e0bfe319f43233599f08 WatchSource:0}: Error finding container 758a54206c06f1e19401d39ee89cc04613d63fe9aa07e0bfe319f43233599f08: Status 404 returned error can't find the container with id 758a54206c06f1e19401d39ee89cc04613d63fe9aa07e0bfe319f43233599f08
Jul 18 00:48:09.930342 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.930307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.930530 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.930362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.930530 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.930444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrtd\" (UniqueName: \"kubernetes.io/projected/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-kube-api-access-lwrtd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.930795 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.930772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.930848 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.930828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:09.937897 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:09.937872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrtd\" (UniqueName: \"kubernetes.io/projected/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-kube-api-access-lwrtd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:10.026033 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.026004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:10.155011 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.154927    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"]
Jul 18 00:48:10.159481 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.159463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.184051 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.184023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"]
Jul 18 00:48:10.185791 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.185768    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"]
Jul 18 00:48:10.187097 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:10.187067    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45e873e_7fbe_48bd_bd1a_0f25d09e6af5.slice/crio-bc05d9daf97850a8368996e0b8ce1afd04f4e10c6e7ee18248d49481eb3afbff WatchSource:0}: Error finding container bc05d9daf97850a8368996e0b8ce1afd04f4e10c6e7ee18248d49481eb3afbff: Status 404 returned error can't find the container with id bc05d9daf97850a8368996e0b8ce1afd04f4e10c6e7ee18248d49481eb3afbff
Jul 18 00:48:10.233577 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.233548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjnc\" (UniqueName: \"kubernetes.io/projected/383d5a6b-b7d6-4230-bb31-b53a88a7978e-kube-api-access-mvjnc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.233801 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.233679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.233801 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.233739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.303010 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.302973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8" event={"ID":"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5","Type":"ContainerStarted","Data":"81cd6cd9ed51c00cec9582ab13fc8aab69e16269365e59c1190614d56195047c"}
Jul 18 00:48:10.303213 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.303035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8" event={"ID":"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5","Type":"ContainerStarted","Data":"bc05d9daf97850a8368996e0b8ce1afd04f4e10c6e7ee18248d49481eb3afbff"}
Jul 18 00:48:10.304704 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.304676    2563 generic.go:356] "Generic (PLEG): container finished" podID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerID="f29d72ecd4906f14447a0b28281db2f821d16134209f0b03cfbb8fa9e8ef7c7b" exitCode=0
Jul 18 00:48:10.304820 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.304764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2" event={"ID":"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae","Type":"ContainerDied","Data":"f29d72ecd4906f14447a0b28281db2f821d16134209f0b03cfbb8fa9e8ef7c7b"}
Jul 18 00:48:10.304820 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.304807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2" event={"ID":"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae","Type":"ContainerStarted","Data":"a9e050ee9ae6c72c27fa893006449072054f7b0d344c2ef30ab2fc489ef4d5dc"}
Jul 18 00:48:10.306266 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.306246    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b0e438c-9503-437a-9114-aad368d08d6b" containerID="b08411f81089e232f825ad55e540a0efd481d951498756673b4df6c3731c9c93" exitCode=0
Jul 18 00:48:10.306370 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.306284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n" event={"ID":"7b0e438c-9503-437a-9114-aad368d08d6b","Type":"ContainerDied","Data":"b08411f81089e232f825ad55e540a0efd481d951498756673b4df6c3731c9c93"}
Jul 18 00:48:10.306370 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.306311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n" event={"ID":"7b0e438c-9503-437a-9114-aad368d08d6b","Type":"ContainerStarted","Data":"758a54206c06f1e19401d39ee89cc04613d63fe9aa07e0bfe319f43233599f08"}
Jul 18 00:48:10.335203 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.335160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjnc\" (UniqueName: \"kubernetes.io/projected/383d5a6b-b7d6-4230-bb31-b53a88a7978e-kube-api-access-mvjnc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.335667 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.335334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.335667 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.335396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.335770 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.335711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.335812 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.335777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.343191 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.343167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjnc\" (UniqueName: \"kubernetes.io/projected/383d5a6b-b7d6-4230-bb31-b53a88a7978e-kube-api-access-mvjnc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.469041 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.469005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:10.600619 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:10.600584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"]
Jul 18 00:48:10.603972 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:10.603940    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383d5a6b_b7d6_4230_bb31_b53a88a7978e.slice/crio-2780c8b67df3a5f237b0ac0ea806013074bdae400bf64e48e127f2827cc05186 WatchSource:0}: Error finding container 2780c8b67df3a5f237b0ac0ea806013074bdae400bf64e48e127f2827cc05186: Status 404 returned error can't find the container with id 2780c8b67df3a5f237b0ac0ea806013074bdae400bf64e48e127f2827cc05186
Jul 18 00:48:11.317453 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.317367    2563 generic.go:356] "Generic (PLEG): container finished" podID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerID="81cd6cd9ed51c00cec9582ab13fc8aab69e16269365e59c1190614d56195047c" exitCode=0
Jul 18 00:48:11.317628 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.317450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8" event={"ID":"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5","Type":"ContainerDied","Data":"81cd6cd9ed51c00cec9582ab13fc8aab69e16269365e59c1190614d56195047c"}
Jul 18 00:48:11.319008 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.318984    2563 generic.go:356] "Generic (PLEG): container finished" podID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerID="9afffcf055f97709b7f83abcebc03d89919112016b8cce5477bbe7ecfdf774dc" exitCode=0
Jul 18 00:48:11.319145 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.319011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt" event={"ID":"383d5a6b-b7d6-4230-bb31-b53a88a7978e","Type":"ContainerDied","Data":"9afffcf055f97709b7f83abcebc03d89919112016b8cce5477bbe7ecfdf774dc"}
Jul 18 00:48:11.319145 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.319070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt" event={"ID":"383d5a6b-b7d6-4230-bb31-b53a88a7978e","Type":"ContainerStarted","Data":"2780c8b67df3a5f237b0ac0ea806013074bdae400bf64e48e127f2827cc05186"}
Jul 18 00:48:11.321161 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.321123    2563 generic.go:356] "Generic (PLEG): container finished" podID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerID="96040ece92f9e03540d1cdfed37ceb105a08bf3262f2a228c204283f69bb1f44" exitCode=0
Jul 18 00:48:11.321255 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.321183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2" event={"ID":"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae","Type":"ContainerDied","Data":"96040ece92f9e03540d1cdfed37ceb105a08bf3262f2a228c204283f69bb1f44"}
Jul 18 00:48:11.323091 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.323071    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b0e438c-9503-437a-9114-aad368d08d6b" containerID="21b95f01b5c57ad6f88356b6e3321c0a3782e246dd06a1bc1b79d7e99847a9c0" exitCode=0
Jul 18 00:48:11.323178 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:11.323140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n" event={"ID":"7b0e438c-9503-437a-9114-aad368d08d6b","Type":"ContainerDied","Data":"21b95f01b5c57ad6f88356b6e3321c0a3782e246dd06a1bc1b79d7e99847a9c0"}
Jul 18 00:48:12.331342 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:12.331309    2563 generic.go:356] "Generic (PLEG): container finished" podID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerID="55a2181810432c6bb5e3df5a8fa5e2f519898feda0301cbbe3f9898849f51aaa" exitCode=0
Jul 18 00:48:12.331753 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:12.331389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8" event={"ID":"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5","Type":"ContainerDied","Data":"55a2181810432c6bb5e3df5a8fa5e2f519898feda0301cbbe3f9898849f51aaa"}
Jul 18 00:48:12.333631 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:12.333605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt" event={"ID":"383d5a6b-b7d6-4230-bb31-b53a88a7978e","Type":"ContainerStarted","Data":"c292136c6de19974dd9ea916ca57a69ef4ca07e1f160f829a24cd92979ca0d66"}
Jul 18 00:48:12.335903 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:12.335881    2563 generic.go:356] "Generic (PLEG): container finished" podID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerID="65d235fc10ecae7a1ea4ff64cc196d38b7e165f053e40335463fd114bffc281c" exitCode=0
Jul 18 00:48:12.335991 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:12.335960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2" event={"ID":"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae","Type":"ContainerDied","Data":"65d235fc10ecae7a1ea4ff64cc196d38b7e165f053e40335463fd114bffc281c"}
Jul 18 00:48:12.338258 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:12.338237    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b0e438c-9503-437a-9114-aad368d08d6b" containerID="834384ff1dd9e30c825d7319c0a3f2358564fb06388875dcbd9a83785a077f3a" exitCode=0
Jul 18 00:48:12.338334 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:12.338269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n" event={"ID":"7b0e438c-9503-437a-9114-aad368d08d6b","Type":"ContainerDied","Data":"834384ff1dd9e30c825d7319c0a3f2358564fb06388875dcbd9a83785a077f3a"}
Jul 18 00:48:13.345553 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.345512    2563 generic.go:356] "Generic (PLEG): container finished" podID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerID="fed9c030950e2fcff42cf8e1022a6614d7357b9d5eaf5b3f105ecf222f2ee104" exitCode=0
Jul 18 00:48:13.345955 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.345555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8" event={"ID":"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5","Type":"ContainerDied","Data":"fed9c030950e2fcff42cf8e1022a6614d7357b9d5eaf5b3f105ecf222f2ee104"}
Jul 18 00:48:13.347275 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.347255    2563 generic.go:356] "Generic (PLEG): container finished" podID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerID="c292136c6de19974dd9ea916ca57a69ef4ca07e1f160f829a24cd92979ca0d66" exitCode=0
Jul 18 00:48:13.347367 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.347344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt" event={"ID":"383d5a6b-b7d6-4230-bb31-b53a88a7978e","Type":"ContainerDied","Data":"c292136c6de19974dd9ea916ca57a69ef4ca07e1f160f829a24cd92979ca0d66"}
Jul 18 00:48:13.488332 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.488308    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:13.560974 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.560948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg8fx\" (UniqueName: \"kubernetes.io/projected/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-kube-api-access-vg8fx\") pod \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") "
Jul 18 00:48:13.561134 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.561021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-util\") pod \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") "
Jul 18 00:48:13.561134 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.561079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-bundle\") pod \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\" (UID: \"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae\") "
Jul 18 00:48:13.561566 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.561535    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-bundle" (OuterVolumeSpecName: "bundle") pod "9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" (UID: "9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:48:13.563517 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.563495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-kube-api-access-vg8fx" (OuterVolumeSpecName: "kube-api-access-vg8fx") pod "9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" (UID: "9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae"). InnerVolumeSpecName "kube-api-access-vg8fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:48:13.567144 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.567105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-util" (OuterVolumeSpecName: "util") pod "9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" (UID: "9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:48:13.577479 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.577456    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:13.662346 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.662238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pnbj\" (UniqueName: \"kubernetes.io/projected/7b0e438c-9503-437a-9114-aad368d08d6b-kube-api-access-6pnbj\") pod \"7b0e438c-9503-437a-9114-aad368d08d6b\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") "
Jul 18 00:48:13.662346 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.662292    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-util\") pod \"7b0e438c-9503-437a-9114-aad368d08d6b\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") "
Jul 18 00:48:13.662573 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.662377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-bundle\") pod \"7b0e438c-9503-437a-9114-aad368d08d6b\" (UID: \"7b0e438c-9503-437a-9114-aad368d08d6b\") "
Jul 18 00:48:13.662629 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.662568    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-util\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:13.662629 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.662586    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:13.662629 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.662601    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg8fx\" (UniqueName: \"kubernetes.io/projected/9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae-kube-api-access-vg8fx\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:13.662795 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.662770    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-bundle" (OuterVolumeSpecName: "bundle") pod "7b0e438c-9503-437a-9114-aad368d08d6b" (UID: "7b0e438c-9503-437a-9114-aad368d08d6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:48:13.664376 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.664353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0e438c-9503-437a-9114-aad368d08d6b-kube-api-access-6pnbj" (OuterVolumeSpecName: "kube-api-access-6pnbj") pod "7b0e438c-9503-437a-9114-aad368d08d6b" (UID: "7b0e438c-9503-437a-9114-aad368d08d6b"). InnerVolumeSpecName "kube-api-access-6pnbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:48:13.668996 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.668966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-util" (OuterVolumeSpecName: "util") pod "7b0e438c-9503-437a-9114-aad368d08d6b" (UID: "7b0e438c-9503-437a-9114-aad368d08d6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:48:13.763384 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.763343    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:13.763384 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.763375    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pnbj\" (UniqueName: \"kubernetes.io/projected/7b0e438c-9503-437a-9114-aad368d08d6b-kube-api-access-6pnbj\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:13.763384 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:13.763385    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b0e438c-9503-437a-9114-aad368d08d6b-util\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:14.354634 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.354598    2563 generic.go:356] "Generic (PLEG): container finished" podID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerID="5691db8e2cdcaf7756be415de0e60a3396d6b558f168127cea8b96fcc47992d6" exitCode=0
Jul 18 00:48:14.355089 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.354677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt" event={"ID":"383d5a6b-b7d6-4230-bb31-b53a88a7978e","Type":"ContainerDied","Data":"5691db8e2cdcaf7756be415de0e60a3396d6b558f168127cea8b96fcc47992d6"}
Jul 18 00:48:14.356475 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.356457    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2"
Jul 18 00:48:14.356634 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.356471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2" event={"ID":"9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae","Type":"ContainerDied","Data":"a9e050ee9ae6c72c27fa893006449072054f7b0d344c2ef30ab2fc489ef4d5dc"}
Jul 18 00:48:14.356634 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.356507    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9e050ee9ae6c72c27fa893006449072054f7b0d344c2ef30ab2fc489ef4d5dc"
Jul 18 00:48:14.358569 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.358542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n" event={"ID":"7b0e438c-9503-437a-9114-aad368d08d6b","Type":"ContainerDied","Data":"758a54206c06f1e19401d39ee89cc04613d63fe9aa07e0bfe319f43233599f08"}
Jul 18 00:48:14.358674 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.358588    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758a54206c06f1e19401d39ee89cc04613d63fe9aa07e0bfe319f43233599f08"
Jul 18 00:48:14.358674 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.358550    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n"
Jul 18 00:48:14.489623 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.489594    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:14.570109 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.570049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-bundle\") pod \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") "
Jul 18 00:48:14.570313 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.570127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-util\") pod \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") "
Jul 18 00:48:14.570313 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.570158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwrtd\" (UniqueName: \"kubernetes.io/projected/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-kube-api-access-lwrtd\") pod \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\" (UID: \"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5\") "
Jul 18 00:48:14.570724 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.570668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-bundle" (OuterVolumeSpecName: "bundle") pod "f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" (UID: "f45e873e-7fbe-48bd-bd1a-0f25d09e6af5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:48:14.572221 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.572199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-kube-api-access-lwrtd" (OuterVolumeSpecName: "kube-api-access-lwrtd") pod "f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" (UID: "f45e873e-7fbe-48bd-bd1a-0f25d09e6af5"). InnerVolumeSpecName "kube-api-access-lwrtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:48:14.575628 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.575602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-util" (OuterVolumeSpecName: "util") pod "f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" (UID: "f45e873e-7fbe-48bd-bd1a-0f25d09e6af5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:48:14.671303 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.671260    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:14.671303 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.671291    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-util\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:14.671303 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:14.671299    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwrtd\" (UniqueName: \"kubernetes.io/projected/f45e873e-7fbe-48bd-bd1a-0f25d09e6af5-kube-api-access-lwrtd\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:15.365332 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.365298    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8"
Jul 18 00:48:15.365697 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.365328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8" event={"ID":"f45e873e-7fbe-48bd-bd1a-0f25d09e6af5","Type":"ContainerDied","Data":"bc05d9daf97850a8368996e0b8ce1afd04f4e10c6e7ee18248d49481eb3afbff"}
Jul 18 00:48:15.365697 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.365382    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc05d9daf97850a8368996e0b8ce1afd04f4e10c6e7ee18248d49481eb3afbff"
Jul 18 00:48:15.499436 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.499408    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:15.579352 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.579314    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-bundle\") pod \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") "
Jul 18 00:48:15.579352 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.579358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-util\") pod \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") "
Jul 18 00:48:15.579576 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.579382    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvjnc\" (UniqueName: \"kubernetes.io/projected/383d5a6b-b7d6-4230-bb31-b53a88a7978e-kube-api-access-mvjnc\") pod \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\" (UID: \"383d5a6b-b7d6-4230-bb31-b53a88a7978e\") "
Jul 18 00:48:15.579875 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.579853    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-bundle" (OuterVolumeSpecName: "bundle") pod "383d5a6b-b7d6-4230-bb31-b53a88a7978e" (UID: "383d5a6b-b7d6-4230-bb31-b53a88a7978e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:48:15.581541 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.581513    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383d5a6b-b7d6-4230-bb31-b53a88a7978e-kube-api-access-mvjnc" (OuterVolumeSpecName: "kube-api-access-mvjnc") pod "383d5a6b-b7d6-4230-bb31-b53a88a7978e" (UID: "383d5a6b-b7d6-4230-bb31-b53a88a7978e"). InnerVolumeSpecName "kube-api-access-mvjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:48:15.584771 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.584734    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-util" (OuterVolumeSpecName: "util") pod "383d5a6b-b7d6-4230-bb31-b53a88a7978e" (UID: "383d5a6b-b7d6-4230-bb31-b53a88a7978e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:48:15.680118 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.680007    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:15.680118 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.680037    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/383d5a6b-b7d6-4230-bb31-b53a88a7978e-util\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:15.680118 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:15.680048    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjnc\" (UniqueName: \"kubernetes.io/projected/383d5a6b-b7d6-4230-bb31-b53a88a7978e-kube-api-access-mvjnc\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:48:16.372411 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:16.372370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt" event={"ID":"383d5a6b-b7d6-4230-bb31-b53a88a7978e","Type":"ContainerDied","Data":"2780c8b67df3a5f237b0ac0ea806013074bdae400bf64e48e127f2827cc05186"}
Jul 18 00:48:16.372884 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:16.372430    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2780c8b67df3a5f237b0ac0ea806013074bdae400bf64e48e127f2827cc05186"
Jul 18 00:48:16.372884 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:16.372391    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt"
Jul 18 00:48:24.679959 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.679904    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"]
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680237    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680249    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680261    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerName="util"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680267    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerName="util"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680275    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerName="util"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680280    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerName="util"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680287    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerName="pull"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680292    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerName="pull"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680297    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerName="util"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680302    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerName="util"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680311    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0e438c-9503-437a-9114-aad368d08d6b" containerName="pull"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680316    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0e438c-9503-437a-9114-aad368d08d6b" containerName="pull"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680324    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerName="pull"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680329    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerName="pull"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680335    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680340    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680347    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerName="pull"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680352    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerName="pull"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680360    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680365    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680373    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0e438c-9503-437a-9114-aad368d08d6b" containerName="util"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680378    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0e438c-9503-437a-9114-aad368d08d6b" containerName="util"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680386    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0e438c-9503-437a-9114-aad368d08d6b" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680392    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0e438c-9503-437a-9114-aad368d08d6b" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680436    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0e438c-9503-437a-9114-aad368d08d6b" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680442    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45e873e-7fbe-48bd-bd1a-0f25d09e6af5" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680452    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae" containerName="extract"
Jul 18 00:48:24.680444 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.680458    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="383d5a6b-b7d6-4230-bb31-b53a88a7978e" containerName="extract"
Jul 18 00:48:24.688705 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.688678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:24.691304 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.691278    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8m8md\""
Jul 18 00:48:24.697146 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.697115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"]
Jul 18 00:48:24.751562 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.751527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n797t\" (UniqueName: \"kubernetes.io/projected/2efe3225-5a34-4864-a2dd-f3d93aa36518-kube-api-access-n797t\") pod \"limitador-operator-controller-manager-859d7b78bd-stpvd\" (UID: \"2efe3225-5a34-4864-a2dd-f3d93aa36518\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:24.852187 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.852143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n797t\" (UniqueName: \"kubernetes.io/projected/2efe3225-5a34-4864-a2dd-f3d93aa36518-kube-api-access-n797t\") pod \"limitador-operator-controller-manager-859d7b78bd-stpvd\" (UID: \"2efe3225-5a34-4864-a2dd-f3d93aa36518\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:24.861016 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.860981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n797t\" (UniqueName: \"kubernetes.io/projected/2efe3225-5a34-4864-a2dd-f3d93aa36518-kube-api-access-n797t\") pod \"limitador-operator-controller-manager-859d7b78bd-stpvd\" (UID: \"2efe3225-5a34-4864-a2dd-f3d93aa36518\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:24.998502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:24.998459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:25.132372 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:25.132345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"]
Jul 18 00:48:25.133916 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:25.133887    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efe3225_5a34_4864_a2dd_f3d93aa36518.slice/crio-29eccf33e7380b1419ace271a3daa29cfac348be03c91b906187239f32880765 WatchSource:0}: Error finding container 29eccf33e7380b1419ace271a3daa29cfac348be03c91b906187239f32880765: Status 404 returned error can't find the container with id 29eccf33e7380b1419ace271a3daa29cfac348be03c91b906187239f32880765
Jul 18 00:48:25.415537 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:25.415442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd" event={"ID":"2efe3225-5a34-4864-a2dd-f3d93aa36518","Type":"ContainerStarted","Data":"29eccf33e7380b1419ace271a3daa29cfac348be03c91b906187239f32880765"}
Jul 18 00:48:26.741389 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.741346    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ccbc64d7b-mnttf"]
Jul 18 00:48:26.744824 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.744797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.756322 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.756286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ccbc64d7b-mnttf"]
Jul 18 00:48:26.869216 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.869119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-serving-cert\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.869216 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.869163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-config\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.869216 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.869199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-service-ca\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.869495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.869269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpphw\" (UniqueName: \"kubernetes.io/projected/f3d2adf1-0225-42be-8ba7-c38f60f95234-kube-api-access-zpphw\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.869495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.869304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-trusted-ca-bundle\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.869495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.869331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-oauth-config\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.869495 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.869411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-oauth-serving-cert\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.970523 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.970475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-trusted-ca-bundle\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.970523 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.970526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-oauth-config\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.970782 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.970572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-oauth-serving-cert\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.970782 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.970603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-serving-cert\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.970782 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.970635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-config\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.970782 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.970674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-service-ca\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.970782 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.970722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpphw\" (UniqueName: \"kubernetes.io/projected/f3d2adf1-0225-42be-8ba7-c38f60f95234-kube-api-access-zpphw\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.971438 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.971405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-config\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.971438 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.971421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-oauth-serving-cert\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.971593 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.971422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-service-ca\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.971593 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.971485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d2adf1-0225-42be-8ba7-c38f60f95234-trusted-ca-bundle\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.973624 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.973601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-oauth-config\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.973749 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.973726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d2adf1-0225-42be-8ba7-c38f60f95234-console-serving-cert\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:26.978516 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:26.978489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpphw\" (UniqueName: \"kubernetes.io/projected/f3d2adf1-0225-42be-8ba7-c38f60f95234-kube-api-access-zpphw\") pod \"console-5ccbc64d7b-mnttf\" (UID: \"f3d2adf1-0225-42be-8ba7-c38f60f95234\") " pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:27.054782 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:27.054741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:27.188636 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:27.188592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ccbc64d7b-mnttf"]
Jul 18 00:48:27.190964 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:27.190933    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d2adf1_0225_42be_8ba7_c38f60f95234.slice/crio-70ac8b98769e8fe62ed0dd71414233f13614458c41acb98224a02bdfb3afb1ed WatchSource:0}: Error finding container 70ac8b98769e8fe62ed0dd71414233f13614458c41acb98224a02bdfb3afb1ed: Status 404 returned error can't find the container with id 70ac8b98769e8fe62ed0dd71414233f13614458c41acb98224a02bdfb3afb1ed
Jul 18 00:48:27.429975 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:27.429868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd" event={"ID":"2efe3225-5a34-4864-a2dd-f3d93aa36518","Type":"ContainerStarted","Data":"a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e"}
Jul 18 00:48:27.429975 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:27.429939    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:27.431486 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:27.431462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccbc64d7b-mnttf" event={"ID":"f3d2adf1-0225-42be-8ba7-c38f60f95234","Type":"ContainerStarted","Data":"0def0918f2df1a94f80f3c4ccd17a319d57b04ced26e75cd3f2c55e02bb34293"}
Jul 18 00:48:27.431595 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:27.431500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ccbc64d7b-mnttf" event={"ID":"f3d2adf1-0225-42be-8ba7-c38f60f95234","Type":"ContainerStarted","Data":"70ac8b98769e8fe62ed0dd71414233f13614458c41acb98224a02bdfb3afb1ed"}
Jul 18 00:48:27.447680 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:27.447630    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd" podStartSLOduration=2.050205236 podStartE2EDuration="3.447614708s" totalImagesPullingTime="1.397409472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:48:27.445501868 +0000 UTC m=+477.765479251" watchObservedRunningTime="2026-07-18 00:48:27.447614708 +0000 UTC m=+477.767592091"
Jul 18 00:48:27.462252 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:27.462206    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ccbc64d7b-mnttf" podStartSLOduration=1.462190892 podStartE2EDuration="1.462190892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:48:27.461669377 +0000 UTC m=+477.781646759" watchObservedRunningTime="2026-07-18 00:48:27.462190892 +0000 UTC m=+477.782168274"
Jul 18 00:48:28.104502 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.104455    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"]
Jul 18 00:48:28.107949 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.107928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"
Jul 18 00:48:28.110507 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.110481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sgn9b\""
Jul 18 00:48:28.110644 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.110528    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 18 00:48:28.117994 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.117962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"]
Jul 18 00:48:28.183007 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.182973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcf7\" (UniqueName: \"kubernetes.io/projected/8f62bc9e-a655-41b2-8a0a-19928ddf5013-kube-api-access-fdcf7\") pod \"dns-operator-controller-manager-864dd78c4c-r5pfv\" (UID: \"8f62bc9e-a655-41b2-8a0a-19928ddf5013\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"
Jul 18 00:48:28.283779 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.283737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcf7\" (UniqueName: \"kubernetes.io/projected/8f62bc9e-a655-41b2-8a0a-19928ddf5013-kube-api-access-fdcf7\") pod \"dns-operator-controller-manager-864dd78c4c-r5pfv\" (UID: \"8f62bc9e-a655-41b2-8a0a-19928ddf5013\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"
Jul 18 00:48:28.296827 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.296794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcf7\" (UniqueName: \"kubernetes.io/projected/8f62bc9e-a655-41b2-8a0a-19928ddf5013-kube-api-access-fdcf7\") pod \"dns-operator-controller-manager-864dd78c4c-r5pfv\" (UID: \"8f62bc9e-a655-41b2-8a0a-19928ddf5013\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"
Jul 18 00:48:28.419974 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.419878    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"
Jul 18 00:48:28.557296 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:28.557266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"]
Jul 18 00:48:28.558914 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:28.558878    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f62bc9e_a655_41b2_8a0a_19928ddf5013.slice/crio-9bb966b66bb1021eb1d2a5513269a5787851291d1ae73813ab76daefcdd915c0 WatchSource:0}: Error finding container 9bb966b66bb1021eb1d2a5513269a5787851291d1ae73813ab76daefcdd915c0: Status 404 returned error can't find the container with id 9bb966b66bb1021eb1d2a5513269a5787851291d1ae73813ab76daefcdd915c0
Jul 18 00:48:29.452182 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:29.452133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv" event={"ID":"8f62bc9e-a655-41b2-8a0a-19928ddf5013","Type":"ContainerStarted","Data":"9bb966b66bb1021eb1d2a5513269a5787851291d1ae73813ab76daefcdd915c0"}
Jul 18 00:48:30.450574 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:30.450547    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 18 00:48:31.463026 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:31.462991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv" event={"ID":"8f62bc9e-a655-41b2-8a0a-19928ddf5013","Type":"ContainerStarted","Data":"a1132993b1d310b257ada519ad1e4fe767a839dd497648ca1c4261e1b78f9b15"}
Jul 18 00:48:31.463477 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:31.463189    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"
Jul 18 00:48:31.486872 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:31.486821    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv" podStartSLOduration=1.601470135 podStartE2EDuration="3.486802848s" totalImagesPullingTime="1.885332713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:48:31.485312833 +0000 UTC m=+481.805290237" watchObservedRunningTime="2026-07-18 00:48:31.486802848 +0000 UTC m=+481.806780231"
Jul 18 00:48:33.102445 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.102405    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"]
Jul 18 00:48:33.109985 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.109953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:33.113634 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.113606    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bbc6v\""
Jul 18 00:48:33.118192 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.118167    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"]
Jul 18 00:48:33.227135 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.227089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnq5\" (UniqueName: \"kubernetes.io/projected/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-kube-api-access-slnq5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" (UID: \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:33.227135 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.227136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" (UID: \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:33.328424 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.328379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnq5\" (UniqueName: \"kubernetes.io/projected/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-kube-api-access-slnq5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" (UID: \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:33.328629 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.328429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" (UID: \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:33.328865 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.328844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" (UID: \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:33.342308 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.342274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnq5\" (UniqueName: \"kubernetes.io/projected/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-kube-api-access-slnq5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" (UID: \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:33.420952 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.420855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:33.564473 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:33.564445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"]
Jul 18 00:48:33.566026 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:33.565994    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dbb891_c7a8_49cb_9bc0_1d0f7a6119bc.slice/crio-78da26c794bf85e9b89fd1adbfce1429ea87940cd8faf1c395bf03408c5c2328 WatchSource:0}: Error finding container 78da26c794bf85e9b89fd1adbfce1429ea87940cd8faf1c395bf03408c5c2328: Status 404 returned error can't find the container with id 78da26c794bf85e9b89fd1adbfce1429ea87940cd8faf1c395bf03408c5c2328
Jul 18 00:48:34.481620 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:34.481573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx" event={"ID":"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc","Type":"ContainerStarted","Data":"78da26c794bf85e9b89fd1adbfce1429ea87940cd8faf1c395bf03408c5c2328"}
Jul 18 00:48:36.915628 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:36.915589    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"]
Jul 18 00:48:36.919529 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:36.919499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"
Jul 18 00:48:36.922142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:36.922113    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-h76sd\""
Jul 18 00:48:36.934561 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:36.934527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"]
Jul 18 00:48:36.961376 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:36.961320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kh9\" (UniqueName: \"kubernetes.io/projected/a1faae7b-f155-4cba-93da-dd68e15bb2ba-kube-api-access-s2kh9\") pod \"authorino-operator-75cc8d5cb7-w28jc\" (UID: \"a1faae7b-f155-4cba-93da-dd68e15bb2ba\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"
Jul 18 00:48:37.054904 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.054869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:37.055125 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.054942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:37.060729 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.060699    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:37.061901 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.061868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kh9\" (UniqueName: \"kubernetes.io/projected/a1faae7b-f155-4cba-93da-dd68e15bb2ba-kube-api-access-s2kh9\") pod \"authorino-operator-75cc8d5cb7-w28jc\" (UID: \"a1faae7b-f155-4cba-93da-dd68e15bb2ba\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"
Jul 18 00:48:37.071748 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.071717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kh9\" (UniqueName: \"kubernetes.io/projected/a1faae7b-f155-4cba-93da-dd68e15bb2ba-kube-api-access-s2kh9\") pod \"authorino-operator-75cc8d5cb7-w28jc\" (UID: \"a1faae7b-f155-4cba-93da-dd68e15bb2ba\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"
Jul 18 00:48:37.232573 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.232536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"
Jul 18 00:48:37.376600 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.376553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"]
Jul 18 00:48:37.379997 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:37.379947    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1faae7b_f155_4cba_93da_dd68e15bb2ba.slice/crio-99e4de36c64168c7ed6b5fab42aa395686c9917794473f948230d04c3166620f WatchSource:0}: Error finding container 99e4de36c64168c7ed6b5fab42aa395686c9917794473f948230d04c3166620f: Status 404 returned error can't find the container with id 99e4de36c64168c7ed6b5fab42aa395686c9917794473f948230d04c3166620f
Jul 18 00:48:37.498952 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.498849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc" event={"ID":"a1faae7b-f155-4cba-93da-dd68e15bb2ba","Type":"ContainerStarted","Data":"99e4de36c64168c7ed6b5fab42aa395686c9917794473f948230d04c3166620f"}
Jul 18 00:48:37.503396 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.503366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ccbc64d7b-mnttf"
Jul 18 00:48:37.561379 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:37.561339    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b95cf4c4-cfxsb"]
Jul 18 00:48:38.443505 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:38.443468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:39.514837 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:39.514760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx" event={"ID":"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc","Type":"ContainerStarted","Data":"9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642"}
Jul 18 00:48:39.515320 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:39.514920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:39.545366 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:39.545301    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx" podStartSLOduration=1.458663993 podStartE2EDuration="6.545277903s" totalImagesPullingTime="5.08661391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:48:39.541406557 +0000 UTC m=+489.861383952" watchObservedRunningTime="2026-07-18 00:48:39.545277903 +0000 UTC m=+489.865255289"
Jul 18 00:48:40.521457 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:40.521409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc" event={"ID":"a1faae7b-f155-4cba-93da-dd68e15bb2ba","Type":"ContainerStarted","Data":"ab48d3be79f5aa8a42ad955bd462aa28087817ec597c9cde17456c1a798d4989"}
Jul 18 00:48:40.522162 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:40.521653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"
Jul 18 00:48:40.550311 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:40.550250    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc" podStartSLOduration=1.556595326 podStartE2EDuration="4.550232146s" totalImagesPullingTime="2.99363682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:48:40.548871575 +0000 UTC m=+490.868848971" watchObservedRunningTime="2026-07-18 00:48:40.550232146 +0000 UTC m=+490.870209528"
Jul 18 00:48:41.736247 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.736209    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"]
Jul 18 00:48:41.739950 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.739927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.744909 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.744880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 18 00:48:41.744909 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.744886    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 18 00:48:41.745124 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.744940    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-75dkv\""
Jul 18 00:48:41.753350 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.753316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"]
Jul 18 00:48:41.806896 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.806860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzll\" (UniqueName: \"kubernetes.io/projected/669e83df-f790-4e7c-acdd-da69464bc03f-kube-api-access-jwzll\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.807101 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.806916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669e83df-f790-4e7c-acdd-da69464bc03f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.807101 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.806940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/669e83df-f790-4e7c-acdd-da69464bc03f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.908297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.908258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzll\" (UniqueName: \"kubernetes.io/projected/669e83df-f790-4e7c-acdd-da69464bc03f-kube-api-access-jwzll\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.908508 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.908320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669e83df-f790-4e7c-acdd-da69464bc03f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.908508 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.908340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/669e83df-f790-4e7c-acdd-da69464bc03f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.909264 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.909235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669e83df-f790-4e7c-acdd-da69464bc03f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.911047 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.911021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/669e83df-f790-4e7c-acdd-da69464bc03f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:41.916243 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:41.916221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzll\" (UniqueName: \"kubernetes.io/projected/669e83df-f790-4e7c-acdd-da69464bc03f-kube-api-access-jwzll\") pod \"kuadrant-console-plugin-66b5774985-4nv6z\" (UID: \"669e83df-f790-4e7c-acdd-da69464bc03f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:42.048652 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:42.048560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"
Jul 18 00:48:42.184367 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:42.184338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z"]
Jul 18 00:48:42.186889 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:48:42.186860    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod669e83df_f790_4e7c_acdd_da69464bc03f.slice/crio-aeed2834f9c218006797b1b32054a935c69010336252292313f7467ea7f6f19c WatchSource:0}: Error finding container aeed2834f9c218006797b1b32054a935c69010336252292313f7467ea7f6f19c: Status 404 returned error can't find the container with id aeed2834f9c218006797b1b32054a935c69010336252292313f7467ea7f6f19c
Jul 18 00:48:42.471635 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:42.471597    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-r5pfv"
Jul 18 00:48:42.532806 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:42.532770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z" event={"ID":"669e83df-f790-4e7c-acdd-da69464bc03f","Type":"ContainerStarted","Data":"aeed2834f9c218006797b1b32054a935c69010336252292313f7467ea7f6f19c"}
Jul 18 00:48:50.524853 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:50.524817    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:51.528843 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:51.528800    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w28jc"
Jul 18 00:48:51.898449 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:51.898364    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"]
Jul 18 00:48:51.912901 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:51.912873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:48:51.929103 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:51.929068    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"]
Jul 18 00:48:52.008494 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.008455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bed217a0-1c55-4d8b-ad97-557772e01fe8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" (UID: \"bed217a0-1c55-4d8b-ad97-557772e01fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:48:52.008703 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.008512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994hl\" (UniqueName: \"kubernetes.io/projected/bed217a0-1c55-4d8b-ad97-557772e01fe8-kube-api-access-994hl\") pod \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" (UID: \"bed217a0-1c55-4d8b-ad97-557772e01fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:48:52.109910 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.109858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bed217a0-1c55-4d8b-ad97-557772e01fe8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" (UID: \"bed217a0-1c55-4d8b-ad97-557772e01fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:48:52.109910 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.109919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994hl\" (UniqueName: \"kubernetes.io/projected/bed217a0-1c55-4d8b-ad97-557772e01fe8-kube-api-access-994hl\") pod \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" (UID: \"bed217a0-1c55-4d8b-ad97-557772e01fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:48:52.110379 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.110351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bed217a0-1c55-4d8b-ad97-557772e01fe8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" (UID: \"bed217a0-1c55-4d8b-ad97-557772e01fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:48:52.132658 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.132616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-994hl\" (UniqueName: \"kubernetes.io/projected/bed217a0-1c55-4d8b-ad97-557772e01fe8-kube-api-access-994hl\") pod \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" (UID: \"bed217a0-1c55-4d8b-ad97-557772e01fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:48:52.226101 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.226039    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:48:52.285423 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.285386    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"]
Jul 18 00:48:52.297656 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.297621    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"]
Jul 18 00:48:52.324022 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.323987    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"]
Jul 18 00:48:52.324306 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.324241    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" containerName="manager" containerID="cri-o://9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642" gracePeriod=2
Jul 18 00:48:52.326413 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.326392    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"]
Jul 18 00:48:52.330313 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.330290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:48:52.333852 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.333822    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"]
Jul 18 00:48:52.354290 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.354197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"]
Jul 18 00:48:52.365897 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.365863    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"]
Jul 18 00:48:52.366197 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.366149    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" containerName="manager" containerID="cri-o://a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e" gracePeriod=2
Jul 18 00:48:52.380828 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.380578    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"]
Jul 18 00:48:52.390128 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.390096    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"]
Jul 18 00:48:52.390594 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.390577    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" containerName="manager"
Jul 18 00:48:52.390690 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.390597    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" containerName="manager"
Jul 18 00:48:52.390690 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.390633    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" containerName="manager"
Jul 18 00:48:52.390690 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.390642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" containerName="manager"
Jul 18 00:48:52.390845 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.390724    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" containerName="manager"
Jul 18 00:48:52.390845 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.390765    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" containerName="manager"
Jul 18 00:48:52.394472 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.394212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:48:52.395325 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:48:52.395258    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:52.413178 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.413135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704ebe56-5519-4a4a-9272-7b59ddf7206a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7827t\" (UID: \"704ebe56-5519-4a4a-9272-7b59ddf7206a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:48:52.413369 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.413221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8mh\" (UniqueName: \"kubernetes.io/projected/704ebe56-5519-4a4a-9272-7b59ddf7206a-kube-api-access-ph8mh\") pod \"kuadrant-operator-controller-manager-55467d9f7-7827t\" (UID: \"704ebe56-5519-4a4a-9272-7b59ddf7206a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:48:52.417373 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.417341    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"]
Jul 18 00:48:52.420270 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.420248    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"]
Jul 18 00:48:52.423929 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.423905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"
Jul 18 00:48:52.437403 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:48:52.437178    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stpvd\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:52.437403 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.437329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"]
Jul 18 00:48:52.440043 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:48:52.439990    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:52.470795 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:48:52.470755    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stpvd\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:48:52.473675 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:48:52.473640    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:48:52.514534 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.514442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704ebe56-5519-4a4a-9272-7b59ddf7206a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7827t\" (UID: \"704ebe56-5519-4a4a-9272-7b59ddf7206a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:48:52.514698 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.514532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8mh\" (UniqueName: \"kubernetes.io/projected/704ebe56-5519-4a4a-9272-7b59ddf7206a-kube-api-access-ph8mh\") pod \"kuadrant-operator-controller-manager-55467d9f7-7827t\" (UID: \"704ebe56-5519-4a4a-9272-7b59ddf7206a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:48:52.514698 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.514568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/492f4fba-47e6-4765-8cf2-541f2049fa27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58ggd\" (UID: \"492f4fba-47e6-4765-8cf2-541f2049fa27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:48:52.514698 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.514606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5j7c\" (UniqueName: \"kubernetes.io/projected/5d634b99-8329-4306-b7a4-4987bc6d61fd-kube-api-access-c5j7c\") pod \"limitador-operator-controller-manager-859d7b78bd-x2nfq\" (UID: \"5d634b99-8329-4306-b7a4-4987bc6d61fd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"
Jul 18 00:48:52.514698 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.514642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdq8k\" (UniqueName: \"kubernetes.io/projected/492f4fba-47e6-4765-8cf2-541f2049fa27-kube-api-access-vdq8k\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58ggd\" (UID: \"492f4fba-47e6-4765-8cf2-541f2049fa27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:48:52.514933 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.514910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704ebe56-5519-4a4a-9272-7b59ddf7206a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-7827t\" (UID: \"704ebe56-5519-4a4a-9272-7b59ddf7206a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:48:52.530811 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.530775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8mh\" (UniqueName: \"kubernetes.io/projected/704ebe56-5519-4a4a-9272-7b59ddf7206a-kube-api-access-ph8mh\") pod \"kuadrant-operator-controller-manager-55467d9f7-7827t\" (UID: \"704ebe56-5519-4a4a-9272-7b59ddf7206a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:48:52.616101 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.616032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/492f4fba-47e6-4765-8cf2-541f2049fa27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58ggd\" (UID: \"492f4fba-47e6-4765-8cf2-541f2049fa27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:48:52.616310 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.616122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5j7c\" (UniqueName: \"kubernetes.io/projected/5d634b99-8329-4306-b7a4-4987bc6d61fd-kube-api-access-c5j7c\") pod \"limitador-operator-controller-manager-859d7b78bd-x2nfq\" (UID: \"5d634b99-8329-4306-b7a4-4987bc6d61fd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"
Jul 18 00:48:52.616310 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.616166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdq8k\" (UniqueName: \"kubernetes.io/projected/492f4fba-47e6-4765-8cf2-541f2049fa27-kube-api-access-vdq8k\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58ggd\" (UID: \"492f4fba-47e6-4765-8cf2-541f2049fa27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:48:52.616472 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.616444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/492f4fba-47e6-4765-8cf2-541f2049fa27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58ggd\" (UID: \"492f4fba-47e6-4765-8cf2-541f2049fa27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:48:52.633233 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.633193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5j7c\" (UniqueName: \"kubernetes.io/projected/5d634b99-8329-4306-b7a4-4987bc6d61fd-kube-api-access-c5j7c\") pod \"limitador-operator-controller-manager-859d7b78bd-x2nfq\" (UID: \"5d634b99-8329-4306-b7a4-4987bc6d61fd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"
Jul 18 00:48:52.637775 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.637746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdq8k\" (UniqueName: \"kubernetes.io/projected/492f4fba-47e6-4765-8cf2-541f2049fa27-kube-api-access-vdq8k\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58ggd\" (UID: \"492f4fba-47e6-4765-8cf2-541f2049fa27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:48:52.750973 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.750923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:48:52.759976 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.759720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:48:52.767292 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:48:52.767212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"
Jul 18 00:49:02.586217 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:02.586141    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b95cf4c4-cfxsb" podUID="9af95e73-31d8-4956-a69d-10a389a88844" containerName="console" containerID="cri-o://1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f" gracePeriod=15
Jul 18 00:49:05.073765 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.073519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:49:05.092115 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.092013    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:49:05.096775 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.096531    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:49:05.102849 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.102824    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:49:05.105273 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.105099    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:49:05.107002 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.106965    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stpvd\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:49:05.131572 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.131517    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"]
Jul 18 00:49:05.134653 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:49:05.134621    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d634b99_8329_4306_b7a4_4987bc6d61fd.slice/crio-f219ea77fbffc5f2dc59ac3b0a768049de25e456dc0456f274052acf0d22e6ac WatchSource:0}: Error finding container f219ea77fbffc5f2dc59ac3b0a768049de25e456dc0456f274052acf0d22e6ac: Status 404 returned error can't find the container with id f219ea77fbffc5f2dc59ac3b0a768049de25e456dc0456f274052acf0d22e6ac
Jul 18 00:49:05.157434 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.157406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"]
Jul 18 00:49:05.158906 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:49:05.158871    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704ebe56_5519_4a4a_9272_7b59ddf7206a.slice/crio-78a1d303d0c4e277a7dda552b51dbe28fc201073297c7f818b0ca7a2412288d1 WatchSource:0}: Error finding container 78a1d303d0c4e277a7dda552b51dbe28fc201073297c7f818b0ca7a2412288d1: Status 404 returned error can't find the container with id 78a1d303d0c4e277a7dda552b51dbe28fc201073297c7f818b0ca7a2412288d1
Jul 18 00:49:05.177290 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.177241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"]
Jul 18 00:49:05.187185 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:49:05.187158    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492f4fba_47e6_4765_8cf2_541f2049fa27.slice/crio-362d0da84352d8989bd43eb4be4ef34e2eacfbad7e03387624edf8286c3f5768 WatchSource:0}: Error finding container 362d0da84352d8989bd43eb4be4ef34e2eacfbad7e03387624edf8286c3f5768: Status 404 returned error can't find the container with id 362d0da84352d8989bd43eb4be4ef34e2eacfbad7e03387624edf8286c3f5768
Jul 18 00:49:05.225818 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.225791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-serving-cert\") pod \"9af95e73-31d8-4956-a69d-10a389a88844\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") "
Jul 18 00:49:05.225947 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.225836    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-trusted-ca-bundle\") pod \"9af95e73-31d8-4956-a69d-10a389a88844\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") "
Jul 18 00:49:05.225947 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.225872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slnq5\" (UniqueName: \"kubernetes.io/projected/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-kube-api-access-slnq5\") pod \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\" (UID: \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\") "
Jul 18 00:49:05.225947 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.225898    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-extensions-socket-volume\") pod \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\" (UID: \"f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc\") "
Jul 18 00:49:05.225947 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.225939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-service-ca\") pod \"9af95e73-31d8-4956-a69d-10a389a88844\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") "
Jul 18 00:49:05.226173 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.225982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-oauth-config\") pod \"9af95e73-31d8-4956-a69d-10a389a88844\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") "
Jul 18 00:49:05.226173 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-oauth-serving-cert\") pod \"9af95e73-31d8-4956-a69d-10a389a88844\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") "
Jul 18 00:49:05.226173 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-console-config\") pod \"9af95e73-31d8-4956-a69d-10a389a88844\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") "
Jul 18 00:49:05.226173 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qzmq\" (UniqueName: \"kubernetes.io/projected/9af95e73-31d8-4956-a69d-10a389a88844-kube-api-access-2qzmq\") pod \"9af95e73-31d8-4956-a69d-10a389a88844\" (UID: \"9af95e73-31d8-4956-a69d-10a389a88844\") "
Jul 18 00:49:05.226173 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n797t\" (UniqueName: \"kubernetes.io/projected/2efe3225-5a34-4864-a2dd-f3d93aa36518-kube-api-access-n797t\") pod \"2efe3225-5a34-4864-a2dd-f3d93aa36518\" (UID: \"2efe3225-5a34-4864-a2dd-f3d93aa36518\") "
Jul 18 00:49:05.226434 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9af95e73-31d8-4956-a69d-10a389a88844" (UID: "9af95e73-31d8-4956-a69d-10a389a88844"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:49:05.226752 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" (UID: "f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:49:05.226752 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226531    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-trusted-ca-bundle\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.226752 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226585    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9af95e73-31d8-4956-a69d-10a389a88844" (UID: "9af95e73-31d8-4956-a69d-10a389a88844"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:49:05.227088 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.226979    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-service-ca" (OuterVolumeSpecName: "service-ca") pod "9af95e73-31d8-4956-a69d-10a389a88844" (UID: "9af95e73-31d8-4956-a69d-10a389a88844"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:49:05.227164 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.227091    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-console-config" (OuterVolumeSpecName: "console-config") pod "9af95e73-31d8-4956-a69d-10a389a88844" (UID: "9af95e73-31d8-4956-a69d-10a389a88844"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:49:05.228410 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.228383    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9af95e73-31d8-4956-a69d-10a389a88844" (UID: "9af95e73-31d8-4956-a69d-10a389a88844"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:49:05.228538 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.228503    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-kube-api-access-slnq5" (OuterVolumeSpecName: "kube-api-access-slnq5") pod "f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" (UID: "f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc"). InnerVolumeSpecName "kube-api-access-slnq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:49:05.228683 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.228663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efe3225-5a34-4864-a2dd-f3d93aa36518-kube-api-access-n797t" (OuterVolumeSpecName: "kube-api-access-n797t") pod "2efe3225-5a34-4864-a2dd-f3d93aa36518" (UID: "2efe3225-5a34-4864-a2dd-f3d93aa36518"). InnerVolumeSpecName "kube-api-access-n797t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:49:05.228848 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.228830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af95e73-31d8-4956-a69d-10a389a88844-kube-api-access-2qzmq" (OuterVolumeSpecName: "kube-api-access-2qzmq") pod "9af95e73-31d8-4956-a69d-10a389a88844" (UID: "9af95e73-31d8-4956-a69d-10a389a88844"). InnerVolumeSpecName "kube-api-access-2qzmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:49:05.228848 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.228833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9af95e73-31d8-4956-a69d-10a389a88844" (UID: "9af95e73-31d8-4956-a69d-10a389a88844"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:49:05.327430 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327383    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-console-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.327430 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327423    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzmq\" (UniqueName: \"kubernetes.io/projected/9af95e73-31d8-4956-a69d-10a389a88844-kube-api-access-2qzmq\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.327430 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327437    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n797t\" (UniqueName: \"kubernetes.io/projected/2efe3225-5a34-4864-a2dd-f3d93aa36518-kube-api-access-n797t\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.327687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327451    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-serving-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.327687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327463    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slnq5\" (UniqueName: \"kubernetes.io/projected/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-kube-api-access-slnq5\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.327687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327477    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc-extensions-socket-volume\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.327687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327488    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.327687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327501    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af95e73-31d8-4956-a69d-10a389a88844-console-oauth-config\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.327687 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.327511    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af95e73-31d8-4956-a69d-10a389a88844-oauth-serving-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:05.688935 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.688902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t" event={"ID":"704ebe56-5519-4a4a-9272-7b59ddf7206a","Type":"ContainerStarted","Data":"7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707"}
Jul 18 00:49:05.689153 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.688956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t" event={"ID":"704ebe56-5519-4a4a-9272-7b59ddf7206a","Type":"ContainerStarted","Data":"78a1d303d0c4e277a7dda552b51dbe28fc201073297c7f818b0ca7a2412288d1"}
Jul 18 00:49:05.689153 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.688990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:49:05.690286 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.690265    2563 generic.go:356] "Generic (PLEG): container finished" podID="2efe3225-5a34-4864-a2dd-f3d93aa36518" containerID="a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e" exitCode=0
Jul 18 00:49:05.690368 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.690303    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:49:05.690368 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.690340    2563 scope.go:117] "RemoveContainer" containerID="a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e"
Jul 18 00:49:05.692190 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.692160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd" event={"ID":"492f4fba-47e6-4765-8cf2-541f2049fa27","Type":"ContainerStarted","Data":"8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960"}
Jul 18 00:49:05.692297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.692208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd" event={"ID":"492f4fba-47e6-4765-8cf2-541f2049fa27","Type":"ContainerStarted","Data":"362d0da84352d8989bd43eb4be4ef34e2eacfbad7e03387624edf8286c3f5768"}
Jul 18 00:49:05.692297 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.692273    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:49:05.693991 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.693954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq" event={"ID":"5d634b99-8329-4306-b7a4-4987bc6d61fd","Type":"ContainerStarted","Data":"6223cee9f473d3faf1a63b110b8c494a8510d699c1c36e7c1b84cacdf83fbf61"}
Jul 18 00:49:05.694114 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.693999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq" event={"ID":"5d634b99-8329-4306-b7a4-4987bc6d61fd","Type":"ContainerStarted","Data":"f219ea77fbffc5f2dc59ac3b0a768049de25e456dc0456f274052acf0d22e6ac"}
Jul 18 00:49:05.694114 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.694039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"
Jul 18 00:49:05.695785 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.695740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z" event={"ID":"669e83df-f790-4e7c-acdd-da69464bc03f","Type":"ContainerStarted","Data":"2277113c99413bf12781247c5a2431b4e5c15e9ea82619e6c2990c2219acc354"}
Jul 18 00:49:05.697268 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.697245    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" containerID="9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642" exitCode=0
Jul 18 00:49:05.697343 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.697283    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:49:05.698658 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.698635    2563 generic.go:356] "Generic (PLEG): container finished" podID="9af95e73-31d8-4956-a69d-10a389a88844" containerID="1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f" exitCode=0
Jul 18 00:49:05.698755 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.698677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b95cf4c4-cfxsb" event={"ID":"9af95e73-31d8-4956-a69d-10a389a88844","Type":"ContainerDied","Data":"1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f"}
Jul 18 00:49:05.698755 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.698710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b95cf4c4-cfxsb" event={"ID":"9af95e73-31d8-4956-a69d-10a389a88844","Type":"ContainerDied","Data":"ee98ea2a8b6198fb130e347b4aad934db31e3e79baeda48ca23b90391fffafd6"}
Jul 18 00:49:05.698755 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.698678    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b95cf4c4-cfxsb"
Jul 18 00:49:05.701569 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.701553    2563 scope.go:117] "RemoveContainer" containerID="a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e"
Jul 18 00:49:05.701831 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.701814    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e\": container with ID starting with a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e not found: ID does not exist" containerID="a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e"
Jul 18 00:49:05.701911 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.701837    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e"} err="failed to get container status \"a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e\": rpc error: code = NotFound desc = could not find container \"a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e\": container with ID starting with a8e3eac44d03be700550933ae7a3af06989527bcdb942282d22d3e18b812a34e not found: ID does not exist"
Jul 18 00:49:05.701911 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.701853    2563 scope.go:117] "RemoveContainer" containerID="9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642"
Jul 18 00:49:05.711484 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.711462    2563 scope.go:117] "RemoveContainer" containerID="9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642"
Jul 18 00:49:05.711757 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.711736    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642\": container with ID starting with 9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642 not found: ID does not exist" containerID="9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642"
Jul 18 00:49:05.711828 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.711767    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642"} err="failed to get container status \"9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642\": rpc error: code = NotFound desc = could not find container \"9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642\": container with ID starting with 9ae1e0bbbfd38f8e7db557685292eab99f6278e70713e70189d9d37dbed32642 not found: ID does not exist"
Jul 18 00:49:05.711828 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.711790    2563 scope.go:117] "RemoveContainer" containerID="1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f"
Jul 18 00:49:05.713819 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.713780    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t" podStartSLOduration=13.713768362 podStartE2EDuration="13.713768362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:05.711069121 +0000 UTC m=+516.031046502" watchObservedRunningTime="2026-07-18 00:49:05.713768362 +0000 UTC m=+516.033745745"
Jul 18 00:49:05.714371 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.714346    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:49:05.716426 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.716392    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stpvd\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:49:05.719902 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.719889    2563 scope.go:117] "RemoveContainer" containerID="1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f"
Jul 18 00:49:05.720250 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.720224    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f\": container with ID starting with 1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f not found: ID does not exist" containerID="1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f"
Jul 18 00:49:05.720310 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.720256    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f"} err="failed to get container status \"1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f\": rpc error: code = NotFound desc = could not find container \"1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f\": container with ID starting with 1f469bf8bd3cf9d34611bafac7b7729c5829a5621747e9fabfc5789e3657815f not found: ID does not exist"
Jul 18 00:49:05.734010 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.733985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b95cf4c4-cfxsb"]
Jul 18 00:49:05.738537 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.738515    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b95cf4c4-cfxsb"]
Jul 18 00:49:05.780661 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.779941    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq" podStartSLOduration=13.779922422 podStartE2EDuration="13.779922422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:05.777841951 +0000 UTC m=+516.097819333" watchObservedRunningTime="2026-07-18 00:49:05.779922422 +0000 UTC m=+516.099899805"
Jul 18 00:49:05.780871 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.780698    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd" podStartSLOduration=13.780682219 podStartE2EDuration="13.780682219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:05.760748051 +0000 UTC m=+516.080725428" watchObservedRunningTime="2026-07-18 00:49:05.780682219 +0000 UTC m=+516.100659606"
Jul 18 00:49:05.780871 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.780728    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njcmx\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njcmx"
Jul 18 00:49:05.782831 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:05.782789    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stpvd\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stpvd"
Jul 18 00:49:05.802704 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:05.802644    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4nv6z" podStartSLOduration=2.00927996 podStartE2EDuration="24.802629073s" totalImagesPullingTime="22.793349113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:05.802177916 +0000 UTC m=+516.122155298" watchObservedRunningTime="2026-07-18 00:49:05.802629073 +0000 UTC m=+516.122606455"
Jul 18 00:49:06.175627 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:06.175589    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2efe3225-5a34-4864-a2dd-f3d93aa36518" path="/var/lib/kubelet/pods/2efe3225-5a34-4864-a2dd-f3d93aa36518/volumes"
Jul 18 00:49:06.175980 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:06.175973    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9af95e73-31d8-4956-a69d-10a389a88844" path="/var/lib/kubelet/pods/9af95e73-31d8-4956-a69d-10a389a88844/volumes"
Jul 18 00:49:06.176341 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:06.176327    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc" path="/var/lib/kubelet/pods/f4dbb891-c7a8-49cb-9bc0-1d0f7a6119bc/volumes"
Jul 18 00:49:07.409342 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:07.409295    2563 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-zp59k_kuadrant-system_bed217a0-1c55-4d8b-ad97-557772e01fe8_0(2d4da12762611b1a12eab0f8f321e7b9bdf95a0e0c0788afd8d7ea4a594904d0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-zp59k to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 18 00:49:07.409610 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:07.409370    2563 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-zp59k_kuadrant-system_bed217a0-1c55-4d8b-ad97-557772e01fe8_0(2d4da12762611b1a12eab0f8f321e7b9bdf95a0e0c0788afd8d7ea4a594904d0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-zp59k to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:07.712713 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:07.712674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:07.715072 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:07.715019    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="bed217a0-1c55-4d8b-ad97-557772e01fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:07.721104 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:07.721079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:07.724087 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:07.724049    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="bed217a0-1c55-4d8b-ad97-557772e01fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:07.747879 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:07.747851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-994hl\" (UniqueName: \"kubernetes.io/projected/bed217a0-1c55-4d8b-ad97-557772e01fe8-kube-api-access-994hl\") pod \"bed217a0-1c55-4d8b-ad97-557772e01fe8\" (UID: \"bed217a0-1c55-4d8b-ad97-557772e01fe8\") "
Jul 18 00:49:07.747994 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:07.747888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bed217a0-1c55-4d8b-ad97-557772e01fe8-extensions-socket-volume\") pod \"bed217a0-1c55-4d8b-ad97-557772e01fe8\" (UID: \"bed217a0-1c55-4d8b-ad97-557772e01fe8\") "
Jul 18 00:49:07.748230 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:07.748209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed217a0-1c55-4d8b-ad97-557772e01fe8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bed217a0-1c55-4d8b-ad97-557772e01fe8" (UID: "bed217a0-1c55-4d8b-ad97-557772e01fe8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:49:07.749979 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:07.749950    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed217a0-1c55-4d8b-ad97-557772e01fe8-kube-api-access-994hl" (OuterVolumeSpecName: "kube-api-access-994hl") pod "bed217a0-1c55-4d8b-ad97-557772e01fe8" (UID: "bed217a0-1c55-4d8b-ad97-557772e01fe8"). InnerVolumeSpecName "kube-api-access-994hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:49:07.848528 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:07.848484    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-994hl\" (UniqueName: \"kubernetes.io/projected/bed217a0-1c55-4d8b-ad97-557772e01fe8-kube-api-access-994hl\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:07.848528 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:07.848516    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bed217a0-1c55-4d8b-ad97-557772e01fe8-extensions-socket-volume\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:08.175341 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:08.175259    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bed217a0-1c55-4d8b-ad97-557772e01fe8" path="/var/lib/kubelet/pods/bed217a0-1c55-4d8b-ad97-557772e01fe8/volumes"
Jul 18 00:49:08.718328 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:08.718297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:08.720563 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:08.720532    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="bed217a0-1c55-4d8b-ad97-557772e01fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:08.722563 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:08.722539    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="bed217a0-1c55-4d8b-ad97-557772e01fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:10.179168 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:10.176617    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-zp59k\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="bed217a0-1c55-4d8b-ad97-557772e01fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-zp59k"
Jul 18 00:49:16.709250 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:16.709220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x2nfq"
Jul 18 00:49:16.709758 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:16.709305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:49:16.709758 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:16.709354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:49:16.770632 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:16.770594    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"]
Jul 18 00:49:16.771429 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:16.771395    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd" podUID="492f4fba-47e6-4765-8cf2-541f2049fa27" containerName="manager" containerID="cri-o://8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960" gracePeriod=10
Jul 18 00:49:17.014327 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.014298    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:49:17.117640 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.117597    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"]
Jul 18 00:49:17.117969 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.117956    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af95e73-31d8-4956-a69d-10a389a88844" containerName="console"
Jul 18 00:49:17.118010 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.117970    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af95e73-31d8-4956-a69d-10a389a88844" containerName="console"
Jul 18 00:49:17.118010 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.117992    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492f4fba-47e6-4765-8cf2-541f2049fa27" containerName="manager"
Jul 18 00:49:17.118010 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.117997    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="492f4fba-47e6-4765-8cf2-541f2049fa27" containerName="manager"
Jul 18 00:49:17.118143 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.118070    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af95e73-31d8-4956-a69d-10a389a88844" containerName="console"
Jul 18 00:49:17.118143 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.118080    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="492f4fba-47e6-4765-8cf2-541f2049fa27" containerName="manager"
Jul 18 00:49:17.120270 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.120246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.132483 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.132455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"]
Jul 18 00:49:17.215169 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.215132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdq8k\" (UniqueName: \"kubernetes.io/projected/492f4fba-47e6-4765-8cf2-541f2049fa27-kube-api-access-vdq8k\") pod \"492f4fba-47e6-4765-8cf2-541f2049fa27\" (UID: \"492f4fba-47e6-4765-8cf2-541f2049fa27\") "
Jul 18 00:49:17.215376 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.215230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/492f4fba-47e6-4765-8cf2-541f2049fa27-extensions-socket-volume\") pod \"492f4fba-47e6-4765-8cf2-541f2049fa27\" (UID: \"492f4fba-47e6-4765-8cf2-541f2049fa27\") "
Jul 18 00:49:17.215376 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.215331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85df70d5-8484-48f8-8461-65fae72d8a96-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vm895\" (UID: \"85df70d5-8484-48f8-8461-65fae72d8a96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.215376 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.215366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66dwk\" (UniqueName: \"kubernetes.io/projected/85df70d5-8484-48f8-8461-65fae72d8a96-kube-api-access-66dwk\") pod \"kuadrant-operator-controller-manager-86878c769f-vm895\" (UID: \"85df70d5-8484-48f8-8461-65fae72d8a96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.215600 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.215572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492f4fba-47e6-4765-8cf2-541f2049fa27-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "492f4fba-47e6-4765-8cf2-541f2049fa27" (UID: "492f4fba-47e6-4765-8cf2-541f2049fa27"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:49:17.217278 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.217258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492f4fba-47e6-4765-8cf2-541f2049fa27-kube-api-access-vdq8k" (OuterVolumeSpecName: "kube-api-access-vdq8k") pod "492f4fba-47e6-4765-8cf2-541f2049fa27" (UID: "492f4fba-47e6-4765-8cf2-541f2049fa27"). InnerVolumeSpecName "kube-api-access-vdq8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:49:17.316582 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.316474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85df70d5-8484-48f8-8461-65fae72d8a96-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vm895\" (UID: \"85df70d5-8484-48f8-8461-65fae72d8a96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.316582 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.316532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66dwk\" (UniqueName: \"kubernetes.io/projected/85df70d5-8484-48f8-8461-65fae72d8a96-kube-api-access-66dwk\") pod \"kuadrant-operator-controller-manager-86878c769f-vm895\" (UID: \"85df70d5-8484-48f8-8461-65fae72d8a96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.316820 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.316595    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/492f4fba-47e6-4765-8cf2-541f2049fa27-extensions-socket-volume\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:17.316820 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.316606    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdq8k\" (UniqueName: \"kubernetes.io/projected/492f4fba-47e6-4765-8cf2-541f2049fa27-kube-api-access-vdq8k\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:17.316913 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.316855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85df70d5-8484-48f8-8461-65fae72d8a96-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vm895\" (UID: \"85df70d5-8484-48f8-8461-65fae72d8a96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.325223 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.325190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66dwk\" (UniqueName: \"kubernetes.io/projected/85df70d5-8484-48f8-8461-65fae72d8a96-kube-api-access-66dwk\") pod \"kuadrant-operator-controller-manager-86878c769f-vm895\" (UID: \"85df70d5-8484-48f8-8461-65fae72d8a96\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.430117 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.430042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.568737 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.568704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"]
Jul 18 00:49:17.570777 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:49:17.570752    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85df70d5_8484_48f8_8461_65fae72d8a96.slice/crio-6643d71ace37364446626c9ee1fd59f58030ad01c685098d5d13a8de2ccfb451 WatchSource:0}: Error finding container 6643d71ace37364446626c9ee1fd59f58030ad01c685098d5d13a8de2ccfb451: Status 404 returned error can't find the container with id 6643d71ace37364446626c9ee1fd59f58030ad01c685098d5d13a8de2ccfb451
Jul 18 00:49:17.779627 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.779589    2563 generic.go:356] "Generic (PLEG): container finished" podID="492f4fba-47e6-4765-8cf2-541f2049fa27" containerID="8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960" exitCode=0
Jul 18 00:49:17.780142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.779640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"
Jul 18 00:49:17.780142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.779672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd" event={"ID":"492f4fba-47e6-4765-8cf2-541f2049fa27","Type":"ContainerDied","Data":"8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960"}
Jul 18 00:49:17.780142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.779747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd" event={"ID":"492f4fba-47e6-4765-8cf2-541f2049fa27","Type":"ContainerDied","Data":"362d0da84352d8989bd43eb4be4ef34e2eacfbad7e03387624edf8286c3f5768"}
Jul 18 00:49:17.780142 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.779786    2563 scope.go:117] "RemoveContainer" containerID="8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960"
Jul 18 00:49:17.781863 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.781815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895" event={"ID":"85df70d5-8484-48f8-8461-65fae72d8a96","Type":"ContainerStarted","Data":"7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0"}
Jul 18 00:49:17.781956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.781878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895" event={"ID":"85df70d5-8484-48f8-8461-65fae72d8a96","Type":"ContainerStarted","Data":"6643d71ace37364446626c9ee1fd59f58030ad01c685098d5d13a8de2ccfb451"}
Jul 18 00:49:17.781956 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.781920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:17.801716 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.801692    2563 scope.go:117] "RemoveContainer" containerID="8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960"
Jul 18 00:49:17.802092 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:17.802070    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960\": container with ID starting with 8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960 not found: ID does not exist" containerID="8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960"
Jul 18 00:49:17.802170 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.802100    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960"} err="failed to get container status \"8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960\": rpc error: code = NotFound desc = could not find container \"8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960\": container with ID starting with 8d80a5b10be9dfbd6e051e1350d04e9233cde8d8a7ad9684960c5665c139e960 not found: ID does not exist"
Jul 18 00:49:17.813855 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.813813    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895" podStartSLOduration=0.813799121 podStartE2EDuration="813.799121ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:17.81155527 +0000 UTC m=+528.131532671" watchObservedRunningTime="2026-07-18 00:49:17.813799121 +0000 UTC m=+528.133776503"
Jul 18 00:49:17.829367 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.829299    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"]
Jul 18 00:49:17.832146 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:17.832117    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58ggd"]
Jul 18 00:49:18.176423 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:18.176344    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492f4fba-47e6-4765-8cf2-541f2049fa27" path="/var/lib/kubelet/pods/492f4fba-47e6-4765-8cf2-541f2049fa27/volumes"
Jul 18 00:49:28.791320 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:28.791240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 00:49:28.840972 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:28.840930    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"]
Jul 18 00:49:28.841405 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:28.841330    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t" podUID="704ebe56-5519-4a4a-9272-7b59ddf7206a" containerName="manager" containerID="cri-o://7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707" gracePeriod=10
Jul 18 00:49:29.147196 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.147160    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:49:29.212836 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.212796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704ebe56-5519-4a4a-9272-7b59ddf7206a-extensions-socket-volume\") pod \"704ebe56-5519-4a4a-9272-7b59ddf7206a\" (UID: \"704ebe56-5519-4a4a-9272-7b59ddf7206a\") "
Jul 18 00:49:29.213039 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.212859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph8mh\" (UniqueName: \"kubernetes.io/projected/704ebe56-5519-4a4a-9272-7b59ddf7206a-kube-api-access-ph8mh\") pod \"704ebe56-5519-4a4a-9272-7b59ddf7206a\" (UID: \"704ebe56-5519-4a4a-9272-7b59ddf7206a\") "
Jul 18 00:49:29.213281 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.213251    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704ebe56-5519-4a4a-9272-7b59ddf7206a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "704ebe56-5519-4a4a-9272-7b59ddf7206a" (UID: "704ebe56-5519-4a4a-9272-7b59ddf7206a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 00:49:29.215019 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.214996    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704ebe56-5519-4a4a-9272-7b59ddf7206a-kube-api-access-ph8mh" (OuterVolumeSpecName: "kube-api-access-ph8mh") pod "704ebe56-5519-4a4a-9272-7b59ddf7206a" (UID: "704ebe56-5519-4a4a-9272-7b59ddf7206a"). InnerVolumeSpecName "kube-api-access-ph8mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:49:29.313683 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.313587    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704ebe56-5519-4a4a-9272-7b59ddf7206a-extensions-socket-volume\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:29.313683 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.313625    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8mh\" (UniqueName: \"kubernetes.io/projected/704ebe56-5519-4a4a-9272-7b59ddf7206a-kube-api-access-ph8mh\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:49:29.853799 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.853761    2563 generic.go:356] "Generic (PLEG): container finished" podID="704ebe56-5519-4a4a-9272-7b59ddf7206a" containerID="7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707" exitCode=0
Jul 18 00:49:29.854201 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.853803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"
Jul 18 00:49:29.854201 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.853818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t" event={"ID":"704ebe56-5519-4a4a-9272-7b59ddf7206a","Type":"ContainerDied","Data":"7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707"}
Jul 18 00:49:29.854201 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.853867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t" event={"ID":"704ebe56-5519-4a4a-9272-7b59ddf7206a","Type":"ContainerDied","Data":"78a1d303d0c4e277a7dda552b51dbe28fc201073297c7f818b0ca7a2412288d1"}
Jul 18 00:49:29.854201 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.853889    2563 scope.go:117] "RemoveContainer" containerID="7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707"
Jul 18 00:49:29.863245 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.863221    2563 scope.go:117] "RemoveContainer" containerID="7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707"
Jul 18 00:49:29.863603 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:49:29.863581    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707\": container with ID starting with 7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707 not found: ID does not exist" containerID="7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707"
Jul 18 00:49:29.863657 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.863613    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707"} err="failed to get container status \"7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707\": rpc error: code = NotFound desc = could not find container \"7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707\": container with ID starting with 7e59976d8d8622d6d14b40c33ed28cefe0d70c14cf3172d965b68c38b32d7707 not found: ID does not exist"
Jul 18 00:49:29.876083 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.876037    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"]
Jul 18 00:49:29.880159 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:29.880132    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-7827t"]
Jul 18 00:49:30.177380 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:30.177307    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704ebe56-5519-4a4a-9272-7b59ddf7206a" path="/var/lib/kubelet/pods/704ebe56-5519-4a4a-9272-7b59ddf7206a/volumes"
Jul 18 00:49:48.699196 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.699155    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"]
Jul 18 00:49:48.699675 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.699504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704ebe56-5519-4a4a-9272-7b59ddf7206a" containerName="manager"
Jul 18 00:49:48.699675 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.699515    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="704ebe56-5519-4a4a-9272-7b59ddf7206a" containerName="manager"
Jul 18 00:49:48.699675 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.699581    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="704ebe56-5519-4a4a-9272-7b59ddf7206a" containerName="manager"
Jul 18 00:49:48.705505 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.705480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:48.708209 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.708188    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 18 00:49:48.712269 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.712244    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"]
Jul 18 00:49:48.793326 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.793288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkgx\" (UniqueName: \"kubernetes.io/projected/2c8e8a98-59c2-484b-96fa-c7b784095b5c-kube-api-access-qkkgx\") pod \"limitador-limitador-64c94fd64f-fgfj4\" (UID: \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:48.793482 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.793355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2c8e8a98-59c2-484b-96fa-c7b784095b5c-config-file\") pod \"limitador-limitador-64c94fd64f-fgfj4\" (UID: \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:48.794605 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.794580    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"]
Jul 18 00:49:48.894900 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.894857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkgx\" (UniqueName: \"kubernetes.io/projected/2c8e8a98-59c2-484b-96fa-c7b784095b5c-kube-api-access-qkkgx\") pod \"limitador-limitador-64c94fd64f-fgfj4\" (UID: \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:48.895135 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.894928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2c8e8a98-59c2-484b-96fa-c7b784095b5c-config-file\") pod \"limitador-limitador-64c94fd64f-fgfj4\" (UID: \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:48.895642 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.895614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2c8e8a98-59c2-484b-96fa-c7b784095b5c-config-file\") pod \"limitador-limitador-64c94fd64f-fgfj4\" (UID: \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:48.902752 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:48.902726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkgx\" (UniqueName: \"kubernetes.io/projected/2c8e8a98-59c2-484b-96fa-c7b784095b5c-kube-api-access-qkkgx\") pod \"limitador-limitador-64c94fd64f-fgfj4\" (UID: \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:49.017598 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.017501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:49.087513 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.087477    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:49:49.091504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.091445    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:49.097540 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.097397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:49:49.132037 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.131998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:49:49.162083 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.162035    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"]
Jul 18 00:49:49.163953 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:49:49.163925    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8e8a98_59c2_484b_96fa_c7b784095b5c.slice/crio-168bfcad61aa16da5a86248060c20b207d32dfb7bff365acffd323d6b1cfa22f WatchSource:0}: Error finding container 168bfcad61aa16da5a86248060c20b207d32dfb7bff365acffd323d6b1cfa22f: Status 404 returned error can't find the container with id 168bfcad61aa16da5a86248060c20b207d32dfb7bff365acffd323d6b1cfa22f
Jul 18 00:49:49.197721 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.197682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f85b8312-3894-4a9e-9a94-10c880a177ea-config-file\") pod \"limitador-limitador-54d6d988d5-2gs9p\" (UID: \"f85b8312-3894-4a9e-9a94-10c880a177ea\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:49.197721 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.197719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrrn\" (UniqueName: \"kubernetes.io/projected/f85b8312-3894-4a9e-9a94-10c880a177ea-kube-api-access-scrrn\") pod \"limitador-limitador-54d6d988d5-2gs9p\" (UID: \"f85b8312-3894-4a9e-9a94-10c880a177ea\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:49.299145 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.299011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f85b8312-3894-4a9e-9a94-10c880a177ea-config-file\") pod \"limitador-limitador-54d6d988d5-2gs9p\" (UID: \"f85b8312-3894-4a9e-9a94-10c880a177ea\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:49.299145 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.299076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrrn\" (UniqueName: \"kubernetes.io/projected/f85b8312-3894-4a9e-9a94-10c880a177ea-kube-api-access-scrrn\") pod \"limitador-limitador-54d6d988d5-2gs9p\" (UID: \"f85b8312-3894-4a9e-9a94-10c880a177ea\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:49.299719 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.299693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f85b8312-3894-4a9e-9a94-10c880a177ea-config-file\") pod \"limitador-limitador-54d6d988d5-2gs9p\" (UID: \"f85b8312-3894-4a9e-9a94-10c880a177ea\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:49.307165 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.307133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrrn\" (UniqueName: \"kubernetes.io/projected/f85b8312-3894-4a9e-9a94-10c880a177ea-kube-api-access-scrrn\") pod \"limitador-limitador-54d6d988d5-2gs9p\" (UID: \"f85b8312-3894-4a9e-9a94-10c880a177ea\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:49.404041 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.403990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:49.548079 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.548036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:49:49.550151 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:49:49.550081    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85b8312_3894_4a9e_9a94_10c880a177ea.slice/crio-cd3bb1d72061e434c9ab203703f0f9f2dba9ba031d85247e9ea8e21c71fe071d WatchSource:0}: Error finding container cd3bb1d72061e434c9ab203703f0f9f2dba9ba031d85247e9ea8e21c71fe071d: Status 404 returned error can't find the container with id cd3bb1d72061e434c9ab203703f0f9f2dba9ba031d85247e9ea8e21c71fe071d
Jul 18 00:49:49.960912 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.960858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4" event={"ID":"2c8e8a98-59c2-484b-96fa-c7b784095b5c","Type":"ContainerStarted","Data":"168bfcad61aa16da5a86248060c20b207d32dfb7bff365acffd323d6b1cfa22f"}
Jul 18 00:49:49.962639 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:49.962599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p" event={"ID":"f85b8312-3894-4a9e-9a94-10c880a177ea","Type":"ContainerStarted","Data":"cd3bb1d72061e434c9ab203703f0f9f2dba9ba031d85247e9ea8e21c71fe071d"}
Jul 18 00:49:51.976247 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:51.976207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4" event={"ID":"2c8e8a98-59c2-484b-96fa-c7b784095b5c","Type":"ContainerStarted","Data":"bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458"}
Jul 18 00:49:51.976875 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:51.976389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:49:51.977781 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:51.977758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p" event={"ID":"f85b8312-3894-4a9e-9a94-10c880a177ea","Type":"ContainerStarted","Data":"fae58e4685aae7260af62a1819256ec49c4ee34f8bde77428ce796872b95f76a"}
Jul 18 00:49:51.977900 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:51.977812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:49:51.993422 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:51.993368    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4" podStartSLOduration=1.312844236 podStartE2EDuration="3.993351901s" totalImagesPullingTime="2.680507665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:51.99114585 +0000 UTC m=+562.311123244" watchObservedRunningTime="2026-07-18 00:49:51.993351901 +0000 UTC m=+562.313329284"
Jul 18 00:49:52.006091 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:49:52.006004    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p" podStartSLOduration=0.70006814 podStartE2EDuration="3.005985957s" totalImagesPullingTime="2.305917817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:52.004763887 +0000 UTC m=+562.324741271" watchObservedRunningTime="2026-07-18 00:49:52.005985957 +0000 UTC m=+562.325963340"
Jul 18 00:50:02.984110 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:02.984040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:50:02.984584 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:02.984158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"
Jul 18 00:50:03.038435 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.038394    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"]
Jul 18 00:50:03.038642 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.038593    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4" podUID="2c8e8a98-59c2-484b-96fa-c7b784095b5c" containerName="limitador" containerID="cri-o://bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458" gracePeriod=30
Jul 18 00:50:03.594242 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.594215    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:50:03.629366 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.629282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkkgx\" (UniqueName: \"kubernetes.io/projected/2c8e8a98-59c2-484b-96fa-c7b784095b5c-kube-api-access-qkkgx\") pod \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\" (UID: \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\") "
Jul 18 00:50:03.629515 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.629380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2c8e8a98-59c2-484b-96fa-c7b784095b5c-config-file\") pod \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\" (UID: \"2c8e8a98-59c2-484b-96fa-c7b784095b5c\") "
Jul 18 00:50:03.629690 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.629667    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8e8a98-59c2-484b-96fa-c7b784095b5c-config-file" (OuterVolumeSpecName: "config-file") pod "2c8e8a98-59c2-484b-96fa-c7b784095b5c" (UID: "2c8e8a98-59c2-484b-96fa-c7b784095b5c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 00:50:03.631412 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.631376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8e8a98-59c2-484b-96fa-c7b784095b5c-kube-api-access-qkkgx" (OuterVolumeSpecName: "kube-api-access-qkkgx") pod "2c8e8a98-59c2-484b-96fa-c7b784095b5c" (UID: "2c8e8a98-59c2-484b-96fa-c7b784095b5c"). InnerVolumeSpecName "kube-api-access-qkkgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:50:03.730005 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.729952    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2c8e8a98-59c2-484b-96fa-c7b784095b5c-config-file\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:50:03.730005 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:03.730002    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkkgx\" (UniqueName: \"kubernetes.io/projected/2c8e8a98-59c2-484b-96fa-c7b784095b5c-kube-api-access-qkkgx\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:50:04.040801 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.040769    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c8e8a98-59c2-484b-96fa-c7b784095b5c" containerID="bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458" exitCode=0
Jul 18 00:50:04.041210 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.040818    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"
Jul 18 00:50:04.041210 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.040857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4" event={"ID":"2c8e8a98-59c2-484b-96fa-c7b784095b5c","Type":"ContainerDied","Data":"bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458"}
Jul 18 00:50:04.041210 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.040911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fgfj4" event={"ID":"2c8e8a98-59c2-484b-96fa-c7b784095b5c","Type":"ContainerDied","Data":"168bfcad61aa16da5a86248060c20b207d32dfb7bff365acffd323d6b1cfa22f"}
Jul 18 00:50:04.041210 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.040936    2563 scope.go:117] "RemoveContainer" containerID="bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458"
Jul 18 00:50:04.049832 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.049815    2563 scope.go:117] "RemoveContainer" containerID="bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458"
Jul 18 00:50:04.050106 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:50:04.050082    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458\": container with ID starting with bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458 not found: ID does not exist" containerID="bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458"
Jul 18 00:50:04.050184 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.050115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458"} err="failed to get container status \"bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458\": rpc error: code = NotFound desc = could not find container \"bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458\": container with ID starting with bb9a65ea5cfc518d279fa38a5e5771d4cae02257e8bb7f36e65b45acc20da458 not found: ID does not exist"
Jul 18 00:50:04.061850 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.061820    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"]
Jul 18 00:50:04.063800 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.063781    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fgfj4"]
Jul 18 00:50:04.176334 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:04.176300    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8e8a98-59c2-484b-96fa-c7b784095b5c" path="/var/lib/kubelet/pods/2c8e8a98-59c2-484b-96fa-c7b784095b5c/volumes"
Jul 18 00:50:24.683154 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.683079    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b4c9f9498-wwql8"]
Jul 18 00:50:24.683765 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.683708    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8e8a98-59c2-484b-96fa-c7b784095b5c" containerName="limitador"
Jul 18 00:50:24.683765 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.683732    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8e8a98-59c2-484b-96fa-c7b784095b5c" containerName="limitador"
Jul 18 00:50:24.683854 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.683804    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8e8a98-59c2-484b-96fa-c7b784095b5c" containerName="limitador"
Jul 18 00:50:24.686816 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.686799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:24.689349 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.689297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 18 00:50:24.689477 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.689374    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zkx4h\""
Jul 18 00:50:24.689477 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.689440    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 18 00:50:24.696512 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.696476    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b4c9f9498-wwql8"]
Jul 18 00:50:24.810366 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.810328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/149498c7-85c8-48a4-ad72-1a6214b71bc4-webhook-cert\") pod \"maas-controller-7b4c9f9498-wwql8\" (UID: \"149498c7-85c8-48a4-ad72-1a6214b71bc4\") " pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:24.810366 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.810368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2w6b\" (UniqueName: \"kubernetes.io/projected/149498c7-85c8-48a4-ad72-1a6214b71bc4-kube-api-access-t2w6b\") pod \"maas-controller-7b4c9f9498-wwql8\" (UID: \"149498c7-85c8-48a4-ad72-1a6214b71bc4\") " pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:24.855321 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.855282    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bc99686f4-6ww29"]
Jul 18 00:50:24.858772 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.858745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:24.871938 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.871908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-6ww29"]
Jul 18 00:50:24.910779 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.910736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl42g\" (UniqueName: \"kubernetes.io/projected/7b464f40-ee0c-4867-8ec1-e317ca57d987-kube-api-access-nl42g\") pod \"maas-controller-5bc99686f4-6ww29\" (UID: \"7b464f40-ee0c-4867-8ec1-e317ca57d987\") " pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:24.910779 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.910784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b464f40-ee0c-4867-8ec1-e317ca57d987-webhook-cert\") pod \"maas-controller-5bc99686f4-6ww29\" (UID: \"7b464f40-ee0c-4867-8ec1-e317ca57d987\") " pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:24.911011 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.910859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/149498c7-85c8-48a4-ad72-1a6214b71bc4-webhook-cert\") pod \"maas-controller-7b4c9f9498-wwql8\" (UID: \"149498c7-85c8-48a4-ad72-1a6214b71bc4\") " pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:24.911011 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.910891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2w6b\" (UniqueName: \"kubernetes.io/projected/149498c7-85c8-48a4-ad72-1a6214b71bc4-kube-api-access-t2w6b\") pod \"maas-controller-7b4c9f9498-wwql8\" (UID: \"149498c7-85c8-48a4-ad72-1a6214b71bc4\") " pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:24.913401 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.913378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/149498c7-85c8-48a4-ad72-1a6214b71bc4-webhook-cert\") pod \"maas-controller-7b4c9f9498-wwql8\" (UID: \"149498c7-85c8-48a4-ad72-1a6214b71bc4\") " pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:24.918565 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.918535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2w6b\" (UniqueName: \"kubernetes.io/projected/149498c7-85c8-48a4-ad72-1a6214b71bc4-kube-api-access-t2w6b\") pod \"maas-controller-7b4c9f9498-wwql8\" (UID: \"149498c7-85c8-48a4-ad72-1a6214b71bc4\") " pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:24.999665 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:24.999631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:25.011994 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:25.011961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl42g\" (UniqueName: \"kubernetes.io/projected/7b464f40-ee0c-4867-8ec1-e317ca57d987-kube-api-access-nl42g\") pod \"maas-controller-5bc99686f4-6ww29\" (UID: \"7b464f40-ee0c-4867-8ec1-e317ca57d987\") " pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:25.012151 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:25.012007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b464f40-ee0c-4867-8ec1-e317ca57d987-webhook-cert\") pod \"maas-controller-5bc99686f4-6ww29\" (UID: \"7b464f40-ee0c-4867-8ec1-e317ca57d987\") " pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:25.014483 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:25.014349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b464f40-ee0c-4867-8ec1-e317ca57d987-webhook-cert\") pod \"maas-controller-5bc99686f4-6ww29\" (UID: \"7b464f40-ee0c-4867-8ec1-e317ca57d987\") " pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:25.020840 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:25.020817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl42g\" (UniqueName: \"kubernetes.io/projected/7b464f40-ee0c-4867-8ec1-e317ca57d987-kube-api-access-nl42g\") pod \"maas-controller-5bc99686f4-6ww29\" (UID: \"7b464f40-ee0c-4867-8ec1-e317ca57d987\") " pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:25.142582 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:25.138477    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b4c9f9498-wwql8"]
Jul 18 00:50:25.152785 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:25.152754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b4c9f9498-wwql8" event={"ID":"149498c7-85c8-48a4-ad72-1a6214b71bc4","Type":"ContainerStarted","Data":"0a0bd34b05895fdfd171d57a70772a99db60238451026c50f12d7f08e0043917"}
Jul 18 00:50:25.169023 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:25.168993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:25.302841 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:25.302811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-6ww29"]
Jul 18 00:50:25.305613 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:50:25.305585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b464f40_ee0c_4867_8ec1_e317ca57d987.slice/crio-653170ddad9ec797b9f1df678a3795220f1108c48a99c0f7ec70dc1f305d518c WatchSource:0}: Error finding container 653170ddad9ec797b9f1df678a3795220f1108c48a99c0f7ec70dc1f305d518c: Status 404 returned error can't find the container with id 653170ddad9ec797b9f1df678a3795220f1108c48a99c0f7ec70dc1f305d518c
Jul 18 00:50:26.164007 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:26.163917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-6ww29" event={"ID":"7b464f40-ee0c-4867-8ec1-e317ca57d987","Type":"ContainerStarted","Data":"653170ddad9ec797b9f1df678a3795220f1108c48a99c0f7ec70dc1f305d518c"}
Jul 18 00:50:28.177616 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:28.177499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-6ww29" event={"ID":"7b464f40-ee0c-4867-8ec1-e317ca57d987","Type":"ContainerStarted","Data":"a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7"}
Jul 18 00:50:28.178130 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:28.177705    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:28.179227 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:28.179191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b4c9f9498-wwql8" event={"ID":"149498c7-85c8-48a4-ad72-1a6214b71bc4","Type":"ContainerStarted","Data":"52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c"}
Jul 18 00:50:28.179359 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:28.179247    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:28.243455 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:28.243404    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bc99686f4-6ww29" podStartSLOduration=1.765951144 podStartE2EDuration="4.243387722s" totalImagesPullingTime="2.477436578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:50:28.241474176 +0000 UTC m=+598.561451561" watchObservedRunningTime="2026-07-18 00:50:28.243387722 +0000 UTC m=+598.563365132"
Jul 18 00:50:28.296020 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:28.295933    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b4c9f9498-wwql8" podStartSLOduration=1.660775176 podStartE2EDuration="4.295914465s" totalImagesPullingTime="2.635139289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:50:28.294223437 +0000 UTC m=+598.614200820" watchObservedRunningTime="2026-07-18 00:50:28.295914465 +0000 UTC m=+598.615891865"
Jul 18 00:50:39.186826 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.186793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:39.189208 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.186865    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:50:39.250628 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.250587    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b4c9f9498-wwql8"]
Jul 18 00:50:39.250823 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.250768    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b4c9f9498-wwql8" podUID="149498c7-85c8-48a4-ad72-1a6214b71bc4" containerName="manager" containerID="cri-o://52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c" gracePeriod=30
Jul 18 00:50:39.518001 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.517972    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:39.644088 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.644013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2w6b\" (UniqueName: \"kubernetes.io/projected/149498c7-85c8-48a4-ad72-1a6214b71bc4-kube-api-access-t2w6b\") pod \"149498c7-85c8-48a4-ad72-1a6214b71bc4\" (UID: \"149498c7-85c8-48a4-ad72-1a6214b71bc4\") "
Jul 18 00:50:39.644301 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.644153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/149498c7-85c8-48a4-ad72-1a6214b71bc4-webhook-cert\") pod \"149498c7-85c8-48a4-ad72-1a6214b71bc4\" (UID: \"149498c7-85c8-48a4-ad72-1a6214b71bc4\") "
Jul 18 00:50:39.646232 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.646197    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149498c7-85c8-48a4-ad72-1a6214b71bc4-kube-api-access-t2w6b" (OuterVolumeSpecName: "kube-api-access-t2w6b") pod "149498c7-85c8-48a4-ad72-1a6214b71bc4" (UID: "149498c7-85c8-48a4-ad72-1a6214b71bc4"). InnerVolumeSpecName "kube-api-access-t2w6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:50:39.646336 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.646242    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149498c7-85c8-48a4-ad72-1a6214b71bc4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "149498c7-85c8-48a4-ad72-1a6214b71bc4" (UID: "149498c7-85c8-48a4-ad72-1a6214b71bc4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:50:39.745188 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.745148    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/149498c7-85c8-48a4-ad72-1a6214b71bc4-webhook-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:50:39.745188 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:39.745184    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2w6b\" (UniqueName: \"kubernetes.io/projected/149498c7-85c8-48a4-ad72-1a6214b71bc4-kube-api-access-t2w6b\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:50:40.244215 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.244181    2563 generic.go:356] "Generic (PLEG): container finished" podID="149498c7-85c8-48a4-ad72-1a6214b71bc4" containerID="52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c" exitCode=0
Jul 18 00:50:40.244708 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.244230    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b4c9f9498-wwql8"
Jul 18 00:50:40.244708 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.244263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b4c9f9498-wwql8" event={"ID":"149498c7-85c8-48a4-ad72-1a6214b71bc4","Type":"ContainerDied","Data":"52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c"}
Jul 18 00:50:40.244708 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.244316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b4c9f9498-wwql8" event={"ID":"149498c7-85c8-48a4-ad72-1a6214b71bc4","Type":"ContainerDied","Data":"0a0bd34b05895fdfd171d57a70772a99db60238451026c50f12d7f08e0043917"}
Jul 18 00:50:40.244708 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.244338    2563 scope.go:117] "RemoveContainer" containerID="52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c"
Jul 18 00:50:40.253657 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.253468    2563 scope.go:117] "RemoveContainer" containerID="52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c"
Jul 18 00:50:40.253764 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:50:40.253746    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c\": container with ID starting with 52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c not found: ID does not exist" containerID="52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c"
Jul 18 00:50:40.253816 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.253773    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c"} err="failed to get container status \"52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c\": rpc error: code = NotFound desc = could not find container \"52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c\": container with ID starting with 52c9136762f9b8aa939eba4963c252e3418882c6578efddd82bb70ac44ee311c not found: ID does not exist"
Jul 18 00:50:40.260541 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.260498    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b4c9f9498-wwql8"]
Jul 18 00:50:40.264983 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:40.264959    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b4c9f9498-wwql8"]
Jul 18 00:50:42.175739 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:50:42.175701    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="149498c7-85c8-48a4-ad72-1a6214b71bc4" path="/var/lib/kubelet/pods/149498c7-85c8-48a4-ad72-1a6214b71bc4/volumes"
Jul 18 00:51:00.458935 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.458894    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"]
Jul 18 00:51:00.459452 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.459297    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149498c7-85c8-48a4-ad72-1a6214b71bc4" containerName="manager"
Jul 18 00:51:00.459452 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.459310    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="149498c7-85c8-48a4-ad72-1a6214b71bc4" containerName="manager"
Jul 18 00:51:00.459452 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.459382    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="149498c7-85c8-48a4-ad72-1a6214b71bc4" containerName="manager"
Jul 18 00:51:00.462558 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.462538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.465584 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.465556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 18 00:51:00.465857 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.465838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 00:51:00.466647 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.466629    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 00:51:00.466702 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.466639    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cnx47\""
Jul 18 00:51:00.480396 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.480370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"]
Jul 18 00:51:00.624087 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.624041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa1b620-c2fa-42d5-a241-4106d914b329-maas-api-tls\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.624285 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.624108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa1b620-c2fa-42d5-a241-4106d914b329-openshift-service-ca\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.624285 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.624173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2f2w\" (UniqueName: \"kubernetes.io/projected/eaa1b620-c2fa-42d5-a241-4106d914b329-kube-api-access-k2f2w\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.725440 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.725358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa1b620-c2fa-42d5-a241-4106d914b329-maas-api-tls\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.725440 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.725421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa1b620-c2fa-42d5-a241-4106d914b329-openshift-service-ca\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.725623 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.725473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2f2w\" (UniqueName: \"kubernetes.io/projected/eaa1b620-c2fa-42d5-a241-4106d914b329-kube-api-access-k2f2w\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.726230 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.726206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa1b620-c2fa-42d5-a241-4106d914b329-openshift-service-ca\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.727841 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.727813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa1b620-c2fa-42d5-a241-4106d914b329-maas-api-tls\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.733548 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.733517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2f2w\" (UniqueName: \"kubernetes.io/projected/eaa1b620-c2fa-42d5-a241-4106d914b329-kube-api-access-k2f2w\") pod \"maas-api-7f9fb8f54-4hqms\" (UID: \"eaa1b620-c2fa-42d5-a241-4106d914b329\") " pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.773140 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.773099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:00.925426 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.925400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"]
Jul 18 00:51:00.927393 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:51:00.927349    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa1b620_c2fa_42d5_a241_4106d914b329.slice/crio-108d6753dde6c8639280121c8bf556e0454806cae4af82e4c0b411fa89e5013f WatchSource:0}: Error finding container 108d6753dde6c8639280121c8bf556e0454806cae4af82e4c0b411fa89e5013f: Status 404 returned error can't find the container with id 108d6753dde6c8639280121c8bf556e0454806cae4af82e4c0b411fa89e5013f
Jul 18 00:51:00.928588 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:00.928568    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 00:51:01.012185 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.012099    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-z5wd9"]
Jul 18 00:51:01.017318 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.017289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:01.020143 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.020111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-phsvb\""
Jul 18 00:51:01.020280 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.020265    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 18 00:51:01.035739 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.030387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-z5wd9"]
Jul 18 00:51:01.126172 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.126133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:51:01.128753 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.128724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8b1761-aa3c-4913-a140-af8dbe0ccb82-plugins-config-volume\") pod \"payload-processing-5bf47cd988-z5wd9\" (UID: \"1d8b1761-aa3c-4913-a140-af8dbe0ccb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:01.128889 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.128770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctf45\" (UniqueName: \"kubernetes.io/projected/1d8b1761-aa3c-4913-a140-af8dbe0ccb82-kube-api-access-ctf45\") pod \"payload-processing-5bf47cd988-z5wd9\" (UID: \"1d8b1761-aa3c-4913-a140-af8dbe0ccb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:01.229666 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.229590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8b1761-aa3c-4913-a140-af8dbe0ccb82-plugins-config-volume\") pod \"payload-processing-5bf47cd988-z5wd9\" (UID: \"1d8b1761-aa3c-4913-a140-af8dbe0ccb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:01.229666 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.229651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctf45\" (UniqueName: \"kubernetes.io/projected/1d8b1761-aa3c-4913-a140-af8dbe0ccb82-kube-api-access-ctf45\") pod \"payload-processing-5bf47cd988-z5wd9\" (UID: \"1d8b1761-aa3c-4913-a140-af8dbe0ccb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:01.230452 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.230424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8b1761-aa3c-4913-a140-af8dbe0ccb82-plugins-config-volume\") pod \"payload-processing-5bf47cd988-z5wd9\" (UID: \"1d8b1761-aa3c-4913-a140-af8dbe0ccb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:01.238486 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.238459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctf45\" (UniqueName: \"kubernetes.io/projected/1d8b1761-aa3c-4913-a140-af8dbe0ccb82-kube-api-access-ctf45\") pod \"payload-processing-5bf47cd988-z5wd9\" (UID: \"1d8b1761-aa3c-4913-a140-af8dbe0ccb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:01.330576 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.330479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:01.362519 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.362482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms" event={"ID":"eaa1b620-c2fa-42d5-a241-4106d914b329","Type":"ContainerStarted","Data":"108d6753dde6c8639280121c8bf556e0454806cae4af82e4c0b411fa89e5013f"}
Jul 18 00:51:01.483592 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:01.483557    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-z5wd9"]
Jul 18 00:51:01.485643 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:51:01.485613    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8b1761_aa3c_4913_a140_af8dbe0ccb82.slice/crio-df8913e20e4e86bc01d819e6eb00eb636c9f9ba5ed8908cbcb93deacd331c780 WatchSource:0}: Error finding container df8913e20e4e86bc01d819e6eb00eb636c9f9ba5ed8908cbcb93deacd331c780: Status 404 returned error can't find the container with id df8913e20e4e86bc01d819e6eb00eb636c9f9ba5ed8908cbcb93deacd331c780
Jul 18 00:51:02.370942 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:02.370865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9" event={"ID":"1d8b1761-aa3c-4913-a140-af8dbe0ccb82","Type":"ContainerStarted","Data":"df8913e20e4e86bc01d819e6eb00eb636c9f9ba5ed8908cbcb93deacd331c780"}
Jul 18 00:51:03.377538 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:03.377431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9" event={"ID":"1d8b1761-aa3c-4913-a140-af8dbe0ccb82","Type":"ContainerStarted","Data":"728a7a1b88dece797dd9c1a738b36e25f10eba7dfc10a4025c450f942b178aa2"}
Jul 18 00:51:03.377984 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:03.377612    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:03.379144 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:03.379117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms" event={"ID":"eaa1b620-c2fa-42d5-a241-4106d914b329","Type":"ContainerStarted","Data":"e10c0b75ce54a6b6748c510d43248665f92b16182c7542530fdbbfb1d3868a57"}
Jul 18 00:51:03.379270 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:03.379250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:03.394711 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:03.394650    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9" podStartSLOduration=1.8250162479999998 podStartE2EDuration="3.394636385s" totalImagesPullingTime="1.569620137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:51:03.393045193 +0000 UTC m=+633.713022577" watchObservedRunningTime="2026-07-18 00:51:03.394636385 +0000 UTC m=+633.714613790"
Jul 18 00:51:03.411333 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:03.411256    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms" podStartSLOduration=1.857598587 podStartE2EDuration="3.41123427s" totalImagesPullingTime="1.553635683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:51:03.407704159 +0000 UTC m=+633.727681556" watchObservedRunningTime="2026-07-18 00:51:03.41123427 +0000 UTC m=+633.731211659"
Jul 18 00:51:09.390022 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:09.389989    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f9fb8f54-4hqms"
Jul 18 00:51:14.387249 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:14.387205    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-z5wd9"
Jul 18 00:51:22.346100 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:22.346039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:51:34.934128 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:34.934084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:51:48.735314 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.735270    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"]
Jul 18 00:51:48.741438 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.741415    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.744194 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.744159    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 18 00:51:48.745347 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.745317    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 18 00:51:48.745347 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.745339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dl528\""
Jul 18 00:51:48.745531 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.745388    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 18 00:51:48.749024 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.748883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"]
Jul 18 00:51:48.823704 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.823660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.823704 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.823708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.823939 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.823728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wf2w\" (UniqueName: \"kubernetes.io/projected/122d0d0b-1459-4f74-aac5-a51282cd3571-kube-api-access-8wf2w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.823939 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.823753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.823939 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.823835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.823939 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.823879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122d0d0b-1459-4f74-aac5-a51282cd3571-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925177 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925177 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925409 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wf2w\" (UniqueName: \"kubernetes.io/projected/122d0d0b-1459-4f74-aac5-a51282cd3571-kube-api-access-8wf2w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925409 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925409 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925409 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122d0d0b-1459-4f74-aac5-a51282cd3571-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925669 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925729 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.925856 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.925836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.927503 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.927481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/122d0d0b-1459-4f74-aac5-a51282cd3571-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.927788 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.927770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122d0d0b-1459-4f74-aac5-a51282cd3571-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:48.932805 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:48.932782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wf2w\" (UniqueName: \"kubernetes.io/projected/122d0d0b-1459-4f74-aac5-a51282cd3571-kube-api-access-8wf2w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq\" (UID: \"122d0d0b-1459-4f74-aac5-a51282cd3571\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:49.053730 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:49.053640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:49.195906 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:49.195875    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"]
Jul 18 00:51:49.197839 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:51:49.197810    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122d0d0b_1459_4f74_aac5_a51282cd3571.slice/crio-6e59482469a7209129cae963165a59ef20334935dc2b831c86ae478d6617fbe4 WatchSource:0}: Error finding container 6e59482469a7209129cae963165a59ef20334935dc2b831c86ae478d6617fbe4: Status 404 returned error can't find the container with id 6e59482469a7209129cae963165a59ef20334935dc2b831c86ae478d6617fbe4
Jul 18 00:51:49.633827 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:49.633782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq" event={"ID":"122d0d0b-1459-4f74-aac5-a51282cd3571","Type":"ContainerStarted","Data":"6e59482469a7209129cae963165a59ef20334935dc2b831c86ae478d6617fbe4"}
Jul 18 00:51:50.234742 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:50.234714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:51:50.641175 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:50.641063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq" event={"ID":"122d0d0b-1459-4f74-aac5-a51282cd3571","Type":"ContainerStarted","Data":"34dc16a62a633fa4eb8a361fb84c91e37b49d50a68cdbb9c756cb27ff43c74b9"}
Jul 18 00:51:50.660318 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:50.660261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq" podStartSLOduration=1.629395013 podStartE2EDuration="2.660244586s" totalImagesPullingTime="1.030849573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:51:50.657484594 +0000 UTC m=+680.977462001" watchObservedRunningTime="2026-07-18 00:51:50.660244586 +0000 UTC m=+680.980221947"
Jul 18 00:51:59.053948 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:59.053904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:59.054491 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:59.053977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:59.066692 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:59.066659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:51:59.701345 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:51:59.701311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq"
Jul 18 00:52:00.336362 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:52:00.336329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:52:07.134293 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:52:07.134255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:52:39.444912 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:52:39.444829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:53:39.645848 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:39.645801    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:53:49.241164 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:49.241125    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:53:56.810009 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:56.809964    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-6ww29"]
Jul 18 00:53:56.810490 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:56.810198    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bc99686f4-6ww29" podUID="7b464f40-ee0c-4867-8ec1-e317ca57d987" containerName="manager" containerID="cri-o://a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7" gracePeriod=30
Jul 18 00:53:57.061210 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.061141    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:53:57.243928 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.243889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl42g\" (UniqueName: \"kubernetes.io/projected/7b464f40-ee0c-4867-8ec1-e317ca57d987-kube-api-access-nl42g\") pod \"7b464f40-ee0c-4867-8ec1-e317ca57d987\" (UID: \"7b464f40-ee0c-4867-8ec1-e317ca57d987\") "
Jul 18 00:53:57.244121 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.243988    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b464f40-ee0c-4867-8ec1-e317ca57d987-webhook-cert\") pod \"7b464f40-ee0c-4867-8ec1-e317ca57d987\" (UID: \"7b464f40-ee0c-4867-8ec1-e317ca57d987\") "
Jul 18 00:53:57.246175 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.246137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b464f40-ee0c-4867-8ec1-e317ca57d987-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7b464f40-ee0c-4867-8ec1-e317ca57d987" (UID: "7b464f40-ee0c-4867-8ec1-e317ca57d987"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:53:57.246319 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.246203    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b464f40-ee0c-4867-8ec1-e317ca57d987-kube-api-access-nl42g" (OuterVolumeSpecName: "kube-api-access-nl42g") pod "7b464f40-ee0c-4867-8ec1-e317ca57d987" (UID: "7b464f40-ee0c-4867-8ec1-e317ca57d987"). InnerVolumeSpecName "kube-api-access-nl42g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:53:57.329533 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.329446    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b464f40-ee0c-4867-8ec1-e317ca57d987" containerID="a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7" exitCode=0
Jul 18 00:53:57.329533 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.329500    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-6ww29"
Jul 18 00:53:57.329714 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.329533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-6ww29" event={"ID":"7b464f40-ee0c-4867-8ec1-e317ca57d987","Type":"ContainerDied","Data":"a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7"}
Jul 18 00:53:57.329714 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.329593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-6ww29" event={"ID":"7b464f40-ee0c-4867-8ec1-e317ca57d987","Type":"ContainerDied","Data":"653170ddad9ec797b9f1df678a3795220f1108c48a99c0f7ec70dc1f305d518c"}
Jul 18 00:53:57.329714 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.329616    2563 scope.go:117] "RemoveContainer" containerID="a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7"
Jul 18 00:53:57.339651 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.339624    2563 scope.go:117] "RemoveContainer" containerID="a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7"
Jul 18 00:53:57.339921 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:53:57.339897    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7\": container with ID starting with a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7 not found: ID does not exist" containerID="a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7"
Jul 18 00:53:57.339970 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.339931    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7"} err="failed to get container status \"a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7\": rpc error: code = NotFound desc = could not find container \"a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7\": container with ID starting with a59e2ee06dda31d16cbbd70f576a7af115d960ae5d7ec4587eacc098dc07f8d7 not found: ID does not exist"
Jul 18 00:53:57.345478 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.345452    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b464f40-ee0c-4867-8ec1-e317ca57d987-webhook-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:53:57.345478 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.345476    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl42g\" (UniqueName: \"kubernetes.io/projected/7b464f40-ee0c-4867-8ec1-e317ca57d987-kube-api-access-nl42g\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:53:57.350826 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.350798    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-6ww29"]
Jul 18 00:53:57.353859 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:57.353836    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-6ww29"]
Jul 18 00:53:58.176014 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:58.175967    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b464f40-ee0c-4867-8ec1-e317ca57d987" path="/var/lib/kubelet/pods/7b464f40-ee0c-4867-8ec1-e317ca57d987/volumes"
Jul 18 00:53:59.647207 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:53:59.647167    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:54:58.137189 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.137138    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bc99686f4-mgn6w"]
Jul 18 00:54:58.137634 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.137515    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b464f40-ee0c-4867-8ec1-e317ca57d987" containerName="manager"
Jul 18 00:54:58.137634 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.137527    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b464f40-ee0c-4867-8ec1-e317ca57d987" containerName="manager"
Jul 18 00:54:58.137634 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.137606    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b464f40-ee0c-4867-8ec1-e317ca57d987" containerName="manager"
Jul 18 00:54:58.140637 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.140617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:58.142880 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.142853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zkx4h\""
Jul 18 00:54:58.143001 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.142923    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 18 00:54:58.143789 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.143764    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 18 00:54:58.148701 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.148680    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-mgn6w"]
Jul 18 00:54:58.256504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.256464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13833ff2-8441-4c85-9d34-10bf386a3304-webhook-cert\") pod \"maas-controller-5bc99686f4-mgn6w\" (UID: \"13833ff2-8441-4c85-9d34-10bf386a3304\") " pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:58.256504 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.256511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s94s\" (UniqueName: \"kubernetes.io/projected/13833ff2-8441-4c85-9d34-10bf386a3304-kube-api-access-8s94s\") pod \"maas-controller-5bc99686f4-mgn6w\" (UID: \"13833ff2-8441-4c85-9d34-10bf386a3304\") " pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:58.357426 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.357383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s94s\" (UniqueName: \"kubernetes.io/projected/13833ff2-8441-4c85-9d34-10bf386a3304-kube-api-access-8s94s\") pod \"maas-controller-5bc99686f4-mgn6w\" (UID: \"13833ff2-8441-4c85-9d34-10bf386a3304\") " pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:58.357613 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.357491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13833ff2-8441-4c85-9d34-10bf386a3304-webhook-cert\") pod \"maas-controller-5bc99686f4-mgn6w\" (UID: \"13833ff2-8441-4c85-9d34-10bf386a3304\") " pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:58.359914 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.359892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13833ff2-8441-4c85-9d34-10bf386a3304-webhook-cert\") pod \"maas-controller-5bc99686f4-mgn6w\" (UID: \"13833ff2-8441-4c85-9d34-10bf386a3304\") " pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:58.365165 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.365138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s94s\" (UniqueName: \"kubernetes.io/projected/13833ff2-8441-4c85-9d34-10bf386a3304-kube-api-access-8s94s\") pod \"maas-controller-5bc99686f4-mgn6w\" (UID: \"13833ff2-8441-4c85-9d34-10bf386a3304\") " pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:58.452100 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.452041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:58.586811 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.586774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-mgn6w"]
Jul 18 00:54:58.587682 ip-10-0-143-142 kubenswrapper[2563]: W0718 00:54:58.587650    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13833ff2_8441_4c85_9d34_10bf386a3304.slice/crio-c0a12784ac6039e511debb1976cfd874fe4c0ce80c53668419bbbdab71ad606c WatchSource:0}: Error finding container c0a12784ac6039e511debb1976cfd874fe4c0ce80c53668419bbbdab71ad606c: Status 404 returned error can't find the container with id c0a12784ac6039e511debb1976cfd874fe4c0ce80c53668419bbbdab71ad606c
Jul 18 00:54:58.660609 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:58.660564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-mgn6w" event={"ID":"13833ff2-8441-4c85-9d34-10bf386a3304","Type":"ContainerStarted","Data":"c0a12784ac6039e511debb1976cfd874fe4c0ce80c53668419bbbdab71ad606c"}
Jul 18 00:54:59.667524 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:59.667486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-mgn6w" event={"ID":"13833ff2-8441-4c85-9d34-10bf386a3304","Type":"ContainerStarted","Data":"7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a"}
Jul 18 00:54:59.667923 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:59.667590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:54:59.685355 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:54:59.685301    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bc99686f4-mgn6w" podStartSLOduration=1.217319984 podStartE2EDuration="1.685281952s" totalImagesPullingTime="467.961968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:54:59.682430604 +0000 UTC m=+870.002407986" watchObservedRunningTime="2026-07-18 00:54:59.685281952 +0000 UTC m=+870.005259352"
Jul 18 00:55:10.674577 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:10.674543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:55:15.287050 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.287007    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-mgn6w"]
Jul 18 00:55:15.287583 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.287367    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bc99686f4-mgn6w" podUID="13833ff2-8441-4c85-9d34-10bf386a3304" containerName="manager" containerID="cri-o://7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a" gracePeriod=30
Jul 18 00:55:15.561498 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.561471    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:55:15.608842 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.608805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13833ff2-8441-4c85-9d34-10bf386a3304-webhook-cert\") pod \"13833ff2-8441-4c85-9d34-10bf386a3304\" (UID: \"13833ff2-8441-4c85-9d34-10bf386a3304\") "
Jul 18 00:55:15.609016 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.608893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s94s\" (UniqueName: \"kubernetes.io/projected/13833ff2-8441-4c85-9d34-10bf386a3304-kube-api-access-8s94s\") pod \"13833ff2-8441-4c85-9d34-10bf386a3304\" (UID: \"13833ff2-8441-4c85-9d34-10bf386a3304\") "
Jul 18 00:55:15.611233 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.611196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13833ff2-8441-4c85-9d34-10bf386a3304-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "13833ff2-8441-4c85-9d34-10bf386a3304" (UID: "13833ff2-8441-4c85-9d34-10bf386a3304"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 00:55:15.611540 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.611517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13833ff2-8441-4c85-9d34-10bf386a3304-kube-api-access-8s94s" (OuterVolumeSpecName: "kube-api-access-8s94s") pod "13833ff2-8441-4c85-9d34-10bf386a3304" (UID: "13833ff2-8441-4c85-9d34-10bf386a3304"). InnerVolumeSpecName "kube-api-access-8s94s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 00:55:15.710369 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.710330    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13833ff2-8441-4c85-9d34-10bf386a3304-webhook-cert\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:55:15.710369 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.710364    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s94s\" (UniqueName: \"kubernetes.io/projected/13833ff2-8441-4c85-9d34-10bf386a3304-kube-api-access-8s94s\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 00:55:15.756245 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.756209    2563 generic.go:356] "Generic (PLEG): container finished" podID="13833ff2-8441-4c85-9d34-10bf386a3304" containerID="7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a" exitCode=0
Jul 18 00:55:15.756381 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.756252    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-mgn6w"
Jul 18 00:55:15.756381 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.756292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-mgn6w" event={"ID":"13833ff2-8441-4c85-9d34-10bf386a3304","Type":"ContainerDied","Data":"7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a"}
Jul 18 00:55:15.756381 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.756343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-mgn6w" event={"ID":"13833ff2-8441-4c85-9d34-10bf386a3304","Type":"ContainerDied","Data":"c0a12784ac6039e511debb1976cfd874fe4c0ce80c53668419bbbdab71ad606c"}
Jul 18 00:55:15.756381 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.756365    2563 scope.go:117] "RemoveContainer" containerID="7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a"
Jul 18 00:55:15.765461 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.765430    2563 scope.go:117] "RemoveContainer" containerID="7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a"
Jul 18 00:55:15.765707 ip-10-0-143-142 kubenswrapper[2563]: E0718 00:55:15.765690    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a\": container with ID starting with 7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a not found: ID does not exist" containerID="7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a"
Jul 18 00:55:15.765752 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.765716    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a"} err="failed to get container status \"7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a\": rpc error: code = NotFound desc = could not find container \"7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a\": container with ID starting with 7fd40df039139edff264066f42103e2f2ee8d9629aff5b745740bb78b5e6bd8a not found: ID does not exist"
Jul 18 00:55:15.776906 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.776882    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-mgn6w"]
Jul 18 00:55:15.780766 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:15.780745    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-mgn6w"]
Jul 18 00:55:16.176491 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.176454    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13833ff2-8441-4c85-9d34-10bf386a3304" path="/var/lib/kubelet/pods/13833ff2-8441-4c85-9d34-10bf386a3304/volumes"
Jul 18 00:55:16.531834 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.531794    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bc99686f4-z7dtd"]
Jul 18 00:55:16.532248 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.532189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13833ff2-8441-4c85-9d34-10bf386a3304" containerName="manager"
Jul 18 00:55:16.532248 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.532201    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="13833ff2-8441-4c85-9d34-10bf386a3304" containerName="manager"
Jul 18 00:55:16.532331 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.532255    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="13833ff2-8441-4c85-9d34-10bf386a3304" containerName="manager"
Jul 18 00:55:16.536871 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.536852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:16.540291 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.540252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 18 00:55:16.540426 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.540325    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 18 00:55:16.540557 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.540539    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zkx4h\""
Jul 18 00:55:16.543408 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.543382    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-z7dtd"]
Jul 18 00:55:16.616440 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.616395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aca6fde9-4e41-4542-b5c0-39461d3782f0-webhook-cert\") pod \"maas-controller-5bc99686f4-z7dtd\" (UID: \"aca6fde9-4e41-4542-b5c0-39461d3782f0\") " pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:16.616628 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.616449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v56rg\" (UniqueName: \"kubernetes.io/projected/aca6fde9-4e41-4542-b5c0-39461d3782f0-kube-api-access-v56rg\") pod \"maas-controller-5bc99686f4-z7dtd\" (UID: \"aca6fde9-4e41-4542-b5c0-39461d3782f0\") " pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:16.717565 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.717526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v56rg\" (UniqueName: \"kubernetes.io/projected/aca6fde9-4e41-4542-b5c0-39461d3782f0-kube-api-access-v56rg\") pod \"maas-controller-5bc99686f4-z7dtd\" (UID: \"aca6fde9-4e41-4542-b5c0-39461d3782f0\") " pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:16.717768 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.717624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aca6fde9-4e41-4542-b5c0-39461d3782f0-webhook-cert\") pod \"maas-controller-5bc99686f4-z7dtd\" (UID: \"aca6fde9-4e41-4542-b5c0-39461d3782f0\") " pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:16.720083 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.720045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aca6fde9-4e41-4542-b5c0-39461d3782f0-webhook-cert\") pod \"maas-controller-5bc99686f4-z7dtd\" (UID: \"aca6fde9-4e41-4542-b5c0-39461d3782f0\") " pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:16.725818 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.725798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v56rg\" (UniqueName: \"kubernetes.io/projected/aca6fde9-4e41-4542-b5c0-39461d3782f0-kube-api-access-v56rg\") pod \"maas-controller-5bc99686f4-z7dtd\" (UID: \"aca6fde9-4e41-4542-b5c0-39461d3782f0\") " pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:16.848644 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.848540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:16.986787 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:16.986762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bc99686f4-z7dtd"]
Jul 18 00:55:17.769777 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:17.769742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-z7dtd" event={"ID":"aca6fde9-4e41-4542-b5c0-39461d3782f0","Type":"ContainerStarted","Data":"dbe6e381cb475af1b29c8e9e635c3bde1f06a35b3e26637ebe1ba8b88f252366"}
Jul 18 00:55:17.770224 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:17.769837    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:17.770224 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:17.769858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bc99686f4-z7dtd" event={"ID":"aca6fde9-4e41-4542-b5c0-39461d3782f0","Type":"ContainerStarted","Data":"d85bf20f3f18aabc7fd396debf8792ab9a10baa2ef428280f95bc5c4d2009c63"}
Jul 18 00:55:17.788254 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:17.788191    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bc99686f4-z7dtd" podStartSLOduration=1.405671452 podStartE2EDuration="1.788174399s" totalImagesPullingTime="382.502947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:55:17.785018958 +0000 UTC m=+888.104996342" watchObservedRunningTime="2026-07-18 00:55:17.788174399 +0000 UTC m=+888.108151815"
Jul 18 00:55:28.778338 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:28.778253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bc99686f4-z7dtd"
Jul 18 00:55:41.939874 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:41.939840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:55:54.234239 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:55:54.234201    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:56:27.843453 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:56:27.843413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:56:39.436211 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:56:39.436177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:56:49.141332 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:56:49.141295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:57:00.935140 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:57:00.935107    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:57:10.439258 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:57:10.439222    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:57:21.236385 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:57:21.236338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:57:32.436633 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:57:32.436585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:57:42.640126 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:57:42.640090    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:58:07.742748 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:58:07.742702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:58:19.589943 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:58:19.589901    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:58:42.740663 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:58:42.740571    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:58:53.638659 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:58:53.638619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:59:08.448651 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:59:08.448616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:59:18.041278 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:59:18.041237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:59:27.832516 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:59:27.832476    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:59:36.941782 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:59:36.941743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 00:59:47.647472 ip-10-0-143-142 kubenswrapper[2563]: I0718 00:59:47.647431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:00:00.148465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:00.148425    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4"]
Jul 18 01:00:00.151910 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:00.151891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4"
Jul 18 01:00:00.154306 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:00.154281    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:00:00.161277 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:00.161257    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4"
Jul 18 01:00:00.167588 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:00.167563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4"]
Jul 18 01:00:00.302506 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:00.302477    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4"]
Jul 18 01:00:00.305155 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:00:00.305122    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4650b9_dd68_43a7_9308_febaaea81688.slice/crio-25e05788e99d62450b7911c161566260e0d2140d5628554053beccd1629675b1 WatchSource:0}: Error finding container 25e05788e99d62450b7911c161566260e0d2140d5628554053beccd1629675b1: Status 404 returned error can't find the container with id 25e05788e99d62450b7911c161566260e0d2140d5628554053beccd1629675b1
Jul 18 01:00:00.307260 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:00.307241    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 01:00:00.348567 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:00.348528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4" event={"ID":"da4650b9-dd68-43a7-9308-febaaea81688","Type":"ContainerStarted","Data":"25e05788e99d62450b7911c161566260e0d2140d5628554053beccd1629675b1"}
Jul 18 01:00:04.380800 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:04.380755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4" event={"ID":"da4650b9-dd68-43a7-9308-febaaea81688","Type":"ContainerStarted","Data":"ca2ae2742fd501134274c665877d1c857ff68f111554bd12220a0be8fb7badfe"}
Jul 18 01:00:04.396835 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:04.396783    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4" podStartSLOduration=0.797210696 podStartE2EDuration="4.396768252s" totalImagesPullingTime="3.599557556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:00:04.395524221 +0000 UTC m=+1174.715501605" watchObservedRunningTime="2026-07-18 01:00:04.396768252 +0000 UTC m=+1174.716745634"
Jul 18 01:00:05.388868 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:05.388831    2563 generic.go:356] "Generic (PLEG): container finished" podID="da4650b9-dd68-43a7-9308-febaaea81688" containerID="ca2ae2742fd501134274c665877d1c857ff68f111554bd12220a0be8fb7badfe" exitCode=0
Jul 18 01:00:05.389285 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:05.388873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4" event={"ID":"da4650b9-dd68-43a7-9308-febaaea81688","Type":"ContainerDied","Data":"ca2ae2742fd501134274c665877d1c857ff68f111554bd12220a0be8fb7badfe"}
Jul 18 01:00:06.539771 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:06.539742    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4"
Jul 18 01:00:07.401347 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:07.401308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4" event={"ID":"da4650b9-dd68-43a7-9308-febaaea81688","Type":"ContainerDied","Data":"25e05788e99d62450b7911c161566260e0d2140d5628554053beccd1629675b1"}
Jul 18 01:00:07.401347 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:07.401332    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738940-h58q4"
Jul 18 01:00:07.401560 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:07.401363    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e05788e99d62450b7911c161566260e0d2140d5628554053beccd1629675b1"
Jul 18 01:00:12.564987 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:12.564950    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"]
Jul 18 01:00:12.565417 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:12.565212    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895" podUID="85df70d5-8484-48f8-8461-65fae72d8a96" containerName="manager" containerID="cri-o://7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0" gracePeriod=10
Jul 18 01:00:13.026393 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.026361    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 01:00:13.160636 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.160524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85df70d5-8484-48f8-8461-65fae72d8a96-extensions-socket-volume\") pod \"85df70d5-8484-48f8-8461-65fae72d8a96\" (UID: \"85df70d5-8484-48f8-8461-65fae72d8a96\") "
Jul 18 01:00:13.160800 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.160652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66dwk\" (UniqueName: \"kubernetes.io/projected/85df70d5-8484-48f8-8461-65fae72d8a96-kube-api-access-66dwk\") pod \"85df70d5-8484-48f8-8461-65fae72d8a96\" (UID: \"85df70d5-8484-48f8-8461-65fae72d8a96\") "
Jul 18 01:00:13.160942 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.160918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85df70d5-8484-48f8-8461-65fae72d8a96-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "85df70d5-8484-48f8-8461-65fae72d8a96" (UID: "85df70d5-8484-48f8-8461-65fae72d8a96"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 01:00:13.162667 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.162638    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85df70d5-8484-48f8-8461-65fae72d8a96-kube-api-access-66dwk" (OuterVolumeSpecName: "kube-api-access-66dwk") pod "85df70d5-8484-48f8-8461-65fae72d8a96" (UID: "85df70d5-8484-48f8-8461-65fae72d8a96"). InnerVolumeSpecName "kube-api-access-66dwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:00:13.261808 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.261754    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66dwk\" (UniqueName: \"kubernetes.io/projected/85df70d5-8484-48f8-8461-65fae72d8a96-kube-api-access-66dwk\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:00:13.261808 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.261800    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85df70d5-8484-48f8-8461-65fae72d8a96-extensions-socket-volume\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:00:13.435508 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.435421    2563 generic.go:356] "Generic (PLEG): container finished" podID="85df70d5-8484-48f8-8461-65fae72d8a96" containerID="7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0" exitCode=0
Jul 18 01:00:13.435508 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.435466    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"
Jul 18 01:00:13.435508 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.435486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895" event={"ID":"85df70d5-8484-48f8-8461-65fae72d8a96","Type":"ContainerDied","Data":"7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0"}
Jul 18 01:00:13.435726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.435531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895" event={"ID":"85df70d5-8484-48f8-8461-65fae72d8a96","Type":"ContainerDied","Data":"6643d71ace37364446626c9ee1fd59f58030ad01c685098d5d13a8de2ccfb451"}
Jul 18 01:00:13.435726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.435553    2563 scope.go:117] "RemoveContainer" containerID="7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0"
Jul 18 01:00:13.446654 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.446630    2563 scope.go:117] "RemoveContainer" containerID="7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0"
Jul 18 01:00:13.447018 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:00:13.446997    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0\": container with ID starting with 7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0 not found: ID does not exist" containerID="7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0"
Jul 18 01:00:13.447112 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.447029    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0"} err="failed to get container status \"7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0\": rpc error: code = NotFound desc = could not find container \"7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0\": container with ID starting with 7760dffcc85f5adb69acaa1c7c390e62ece2d7acd7a52ca611aeb37aabfdcee0 not found: ID does not exist"
Jul 18 01:00:13.457864 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.457837    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"]
Jul 18 01:00:13.460988 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:13.460963    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vm895"]
Jul 18 01:00:14.175722 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:00:14.175675    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85df70d5-8484-48f8-8461-65fae72d8a96" path="/var/lib/kubelet/pods/85df70d5-8484-48f8-8461-65fae72d8a96/volumes"
Jul 18 01:01:13.696773 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.696735    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"]
Jul 18 01:01:13.697262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.697119    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85df70d5-8484-48f8-8461-65fae72d8a96" containerName="manager"
Jul 18 01:01:13.697262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.697131    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85df70d5-8484-48f8-8461-65fae72d8a96" containerName="manager"
Jul 18 01:01:13.697262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.697142    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4650b9-dd68-43a7-9308-febaaea81688" containerName="cleanup"
Jul 18 01:01:13.697262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.697148    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4650b9-dd68-43a7-9308-febaaea81688" containerName="cleanup"
Jul 18 01:01:13.697262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.697202    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4650b9-dd68-43a7-9308-febaaea81688" containerName="cleanup"
Jul 18 01:01:13.697262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.697212    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85df70d5-8484-48f8-8461-65fae72d8a96" containerName="manager"
Jul 18 01:01:13.700258 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.700239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:13.703341 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.703315    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bbc6v\""
Jul 18 01:01:13.712425 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.712401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"]
Jul 18 01:01:13.861959 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.861918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwjt7\" (UniqueName: \"kubernetes.io/projected/80dd2b8c-bf37-4c84-8579-4bddc6d9070f-kube-api-access-qwjt7\") pod \"kuadrant-operator-controller-manager-86878c769f-zfbfh\" (UID: \"80dd2b8c-bf37-4c84-8579-4bddc6d9070f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:13.862169 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.862048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80dd2b8c-bf37-4c84-8579-4bddc6d9070f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zfbfh\" (UID: \"80dd2b8c-bf37-4c84-8579-4bddc6d9070f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:13.962873 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.962774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80dd2b8c-bf37-4c84-8579-4bddc6d9070f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zfbfh\" (UID: \"80dd2b8c-bf37-4c84-8579-4bddc6d9070f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:13.962873 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.962834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwjt7\" (UniqueName: \"kubernetes.io/projected/80dd2b8c-bf37-4c84-8579-4bddc6d9070f-kube-api-access-qwjt7\") pod \"kuadrant-operator-controller-manager-86878c769f-zfbfh\" (UID: \"80dd2b8c-bf37-4c84-8579-4bddc6d9070f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:13.963212 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.963190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80dd2b8c-bf37-4c84-8579-4bddc6d9070f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zfbfh\" (UID: \"80dd2b8c-bf37-4c84-8579-4bddc6d9070f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:13.972489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:13.972460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwjt7\" (UniqueName: \"kubernetes.io/projected/80dd2b8c-bf37-4c84-8579-4bddc6d9070f-kube-api-access-qwjt7\") pod \"kuadrant-operator-controller-manager-86878c769f-zfbfh\" (UID: \"80dd2b8c-bf37-4c84-8579-4bddc6d9070f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:14.011499 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:14.011442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:14.159263 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:14.159237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"]
Jul 18 01:01:14.162065 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:01:14.162021    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dd2b8c_bf37_4c84_8579_4bddc6d9070f.slice/crio-23da4ba19e77be0a327fe580f739c9bb27fedd30b1d9a5f10b19b65273cf64b4 WatchSource:0}: Error finding container 23da4ba19e77be0a327fe580f739c9bb27fedd30b1d9a5f10b19b65273cf64b4: Status 404 returned error can't find the container with id 23da4ba19e77be0a327fe580f739c9bb27fedd30b1d9a5f10b19b65273cf64b4
Jul 18 01:01:14.775771 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:14.775734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh" event={"ID":"80dd2b8c-bf37-4c84-8579-4bddc6d9070f","Type":"ContainerStarted","Data":"17d0acc686fa37aadc05601910e8c0f6ff94f5d4f509fd1eb4e6fb36c0371ef9"}
Jul 18 01:01:14.776187 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:14.775885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:14.776187 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:14.775930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh" event={"ID":"80dd2b8c-bf37-4c84-8579-4bddc6d9070f","Type":"ContainerStarted","Data":"23da4ba19e77be0a327fe580f739c9bb27fedd30b1d9a5f10b19b65273cf64b4"}
Jul 18 01:01:14.796868 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:14.796809    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh" podStartSLOduration=1.7967883759999999 podStartE2EDuration="1.796788376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:01:14.795230698 +0000 UTC m=+1245.115208080" watchObservedRunningTime="2026-07-18 01:01:14.796788376 +0000 UTC m=+1245.116765801"
Jul 18 01:01:25.783506 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:25.783430    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zfbfh"
Jul 18 01:01:34.742120 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:34.742080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:01:44.144105 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:44.144064    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:01:57.040753 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:01:57.040712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:02:07.942899 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:02:07.942852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:02:19.134404 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:02:19.134362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:02:29.738023 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:02:29.737989    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:02:39.335760 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:02:39.335718    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:02:48.834915 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:02:48.834880    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:03:00.644255 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:03:00.644216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:03:09.632400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:03:09.632352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:03:22.140241 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:03:22.140204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:03:44.337347 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:03:44.337309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:03:51.847009 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:03:51.846968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:04:01.832227 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:04:01.832184    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:04:12.334011 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:04:12.333971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:04:22.143186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:04:22.143095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:04:33.334089 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:04:33.334030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:04:42.638294 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:04:42.638248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:04:53.135972 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:04:53.135931    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:05:01.239084 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:05:01.239030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:05:10.734226 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:05:10.734192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:05:31.734600 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:05:31.734556    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:05:38.831338 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:05:38.831299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:05:48.454008 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:05:48.453975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:05:57.943032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:05:57.942983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:06:07.742164 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:06:07.742126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:06:20.332505 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:06:20.332466    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:06:30.846618 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:06:30.846578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:06:41.945485 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:06:41.945447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:06:54.738820 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:06:54.738785    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:07:09.233165 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:09.233125    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:07:20.234266 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:20.234221    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:07:34.828030 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:34.827943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:07:43.273239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.273187    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"]
Jul 18 01:07:43.276958 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.276939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.279676 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.279472    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9bd91509-serving-cert\""
Jul 18 01:07:43.286156 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.286123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"]
Jul 18 01:07:43.305073 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.305025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52274670-4bc0-49fe-b98b-f220a2860e5e-openshift-service-ca\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.305251 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.305084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrvzl\" (UniqueName: \"kubernetes.io/projected/52274670-4bc0-49fe-b98b-f220a2860e5e-kube-api-access-hrvzl\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.305251 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.305174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.405735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.405688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52274670-4bc0-49fe-b98b-f220a2860e5e-openshift-service-ca\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.405735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.405738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrvzl\" (UniqueName: \"kubernetes.io/projected/52274670-4bc0-49fe-b98b-f220a2860e5e-kube-api-access-hrvzl\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.406032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.405784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.406032 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:07:43.405895    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-serving-cert: secret "maas-api-e2e-ait-9bd91509-serving-cert" not found
Jul 18 01:07:43.406032 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:07:43.405962    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls podName:52274670-4bc0-49fe-b98b-f220a2860e5e nodeName:}" failed. No retries permitted until 2026-07-18 01:07:43.905943271 +0000 UTC m=+1634.225920633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls") pod "maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" (UID: "52274670-4bc0-49fe-b98b-f220a2860e5e") : secret "maas-api-e2e-ait-9bd91509-serving-cert" not found
Jul 18 01:07:43.406508 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.406481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52274670-4bc0-49fe-b98b-f220a2860e5e-openshift-service-ca\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.416705 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.416672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrvzl\" (UniqueName: \"kubernetes.io/projected/52274670-4bc0-49fe-b98b-f220a2860e5e-kube-api-access-hrvzl\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.908524 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.908477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:43.911265 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:43.911241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls\") pod \"maas-api-e2e-ait-9bd91509-77f8999676-h4z7z\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:44.191958 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:44.191879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:44.339553 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:44.339519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"]
Jul 18 01:07:44.340939 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:07:44.340896    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52274670_4bc0_49fe_b98b_f220a2860e5e.slice/crio-e896883e482b6c7c396c48657dd29274ecbb1673d6ec838072943868dd58f998 WatchSource:0}: Error finding container e896883e482b6c7c396c48657dd29274ecbb1673d6ec838072943868dd58f998: Status 404 returned error can't find the container with id e896883e482b6c7c396c48657dd29274ecbb1673d6ec838072943868dd58f998
Jul 18 01:07:44.342813 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:44.342795    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 01:07:44.920500 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:44.920460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerStarted","Data":"584a69fdf1c8eacb9d7c6abc7ed2484f410a112905f6ac8a723d5d0ae053787d"}
Jul 18 01:07:44.920656 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:44.920518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerStarted","Data":"e896883e482b6c7c396c48657dd29274ecbb1673d6ec838072943868dd58f998"}
Jul 18 01:07:44.920656 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:44.920544    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:44.937625 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:44.937579    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podStartSLOduration=1.454838948 podStartE2EDuration="1.937564572s" totalImagesPullingTime="482.725624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:07:44.935997029 +0000 UTC m=+1635.255974414" watchObservedRunningTime="2026-07-18 01:07:44.937564572 +0000 UTC m=+1635.257541955"
Jul 18 01:07:45.928037 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:45.928002    2563 generic.go:356] "Generic (PLEG): container finished" podID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerID="584a69fdf1c8eacb9d7c6abc7ed2484f410a112905f6ac8a723d5d0ae053787d" exitCode=1
Jul 18 01:07:45.928435 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:45.928083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerDied","Data":"584a69fdf1c8eacb9d7c6abc7ed2484f410a112905f6ac8a723d5d0ae053787d"}
Jul 18 01:07:45.928435 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:45.928361    2563 scope.go:117] "RemoveContainer" containerID="584a69fdf1c8eacb9d7c6abc7ed2484f410a112905f6ac8a723d5d0ae053787d"
Jul 18 01:07:46.442463 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:46.442431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:07:46.935886 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:46.935849    2563 generic.go:356] "Generic (PLEG): container finished" podID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerID="25741f1f5115afd167414dff94800e71ec398bfa100df26b7319965ad1ecf6d3" exitCode=1
Jul 18 01:07:46.936349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:46.935939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerDied","Data":"25741f1f5115afd167414dff94800e71ec398bfa100df26b7319965ad1ecf6d3"}
Jul 18 01:07:46.936349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:46.935999    2563 scope.go:117] "RemoveContainer" containerID="584a69fdf1c8eacb9d7c6abc7ed2484f410a112905f6ac8a723d5d0ae053787d"
Jul 18 01:07:46.936349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:46.936169    2563 scope.go:117] "RemoveContainer" containerID="25741f1f5115afd167414dff94800e71ec398bfa100df26b7319965ad1ecf6d3"
Jul 18 01:07:46.936492 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:07:46.936382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:07:47.943070 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:47.943037    2563 scope.go:117] "RemoveContainer" containerID="25741f1f5115afd167414dff94800e71ec398bfa100df26b7319965ad1ecf6d3"
Jul 18 01:07:47.943467 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:07:47.943239    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:07:49.921406 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:49.921365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:49.921845 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:49.921761    2563 scope.go:117] "RemoveContainer" containerID="25741f1f5115afd167414dff94800e71ec398bfa100df26b7319965ad1ecf6d3"
Jul 18 01:07:49.921957 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:07:49.921937    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:07:54.191982 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:54.191950    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:07:54.192402 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:07:54.192318    2563 scope.go:117] "RemoveContainer" containerID="25741f1f5115afd167414dff94800e71ec398bfa100df26b7319965ad1ecf6d3"
Jul 18 01:07:54.192496 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:07:54.192471    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:08:06.170082 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:06.170012    2563 scope.go:117] "RemoveContainer" containerID="25741f1f5115afd167414dff94800e71ec398bfa100df26b7319965ad1ecf6d3"
Jul 18 01:08:07.047947 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:07.047911    2563 generic.go:356] "Generic (PLEG): container finished" podID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerID="d5797988952a947436e1a5140d6ace2e46752dc8e52ecb49f11210eb846d8414" exitCode=1
Jul 18 01:08:07.048176 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:07.047971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerDied","Data":"d5797988952a947436e1a5140d6ace2e46752dc8e52ecb49f11210eb846d8414"}
Jul 18 01:08:07.048176 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:07.048031    2563 scope.go:117] "RemoveContainer" containerID="25741f1f5115afd167414dff94800e71ec398bfa100df26b7319965ad1ecf6d3"
Jul 18 01:08:07.048438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:07.048420    2563 scope.go:117] "RemoveContainer" containerID="d5797988952a947436e1a5140d6ace2e46752dc8e52ecb49f11210eb846d8414"
Jul 18 01:08:07.048666 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:08:07.048631    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:08:09.921115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:09.921044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:08:09.921568 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:09.921548    2563 scope.go:117] "RemoveContainer" containerID="d5797988952a947436e1a5140d6ace2e46752dc8e52ecb49f11210eb846d8414"
Jul 18 01:08:09.921759 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:08:09.921738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:08:14.192291 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:14.192253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:08:14.192809 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:14.192789    2563 scope.go:117] "RemoveContainer" containerID="d5797988952a947436e1a5140d6ace2e46752dc8e52ecb49f11210eb846d8414"
Jul 18 01:08:14.193012 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:08:14.192989    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:08:29.169800 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:29.169765    2563 scope.go:117] "RemoveContainer" containerID="d5797988952a947436e1a5140d6ace2e46752dc8e52ecb49f11210eb846d8414"
Jul 18 01:08:30.176498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:30.176416    2563 generic.go:356] "Generic (PLEG): container finished" podID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerID="72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995" exitCode=1
Jul 18 01:08:30.176498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:30.176482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerDied","Data":"72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995"}
Jul 18 01:08:30.176949 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:30.176546    2563 scope.go:117] "RemoveContainer" containerID="d5797988952a947436e1a5140d6ace2e46752dc8e52ecb49f11210eb846d8414"
Jul 18 01:08:30.176949 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:30.176840    2563 scope.go:117] "RemoveContainer" containerID="72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995"
Jul 18 01:08:30.177089 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:08:30.177046    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:08:34.192594 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:34.192560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:08:34.192982 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:34.192898    2563 scope.go:117] "RemoveContainer" containerID="72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995"
Jul 18 01:08:34.193128 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:08:34.193106    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:08:34.921271 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:34.921230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:08:34.921667 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:34.921653    2563 scope.go:117] "RemoveContainer" containerID="72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995"
Jul 18 01:08:34.921846 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:08:34.921827    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:08:47.170031 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:08:47.169994    2563 scope.go:117] "RemoveContainer" containerID="72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995"
Jul 18 01:08:47.170552 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:08:47.170215    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:09:00.172321 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:00.172291    2563 scope.go:117] "RemoveContainer" containerID="72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995"
Jul 18 01:09:00.172778 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:09:00.172476    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:09:15.169982 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:15.169941    2563 scope.go:117] "RemoveContainer" containerID="72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995"
Jul 18 01:09:16.438481 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:16.438449    2563 generic.go:356] "Generic (PLEG): container finished" podID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d" exitCode=1
Jul 18 01:09:16.438891 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:16.438517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerDied","Data":"70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"}
Jul 18 01:09:16.438891 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:16.438579    2563 scope.go:117] "RemoveContainer" containerID="72895cc22e03144fc9325b3c952963ea8148054719fa6a8975a50ed4fda6b995"
Jul 18 01:09:16.439015 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:16.438951    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:09:16.439208 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:09:16.439184    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:09:19.920961 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:19.920914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:09:19.921582 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:19.921457    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:09:19.921711 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:09:19.921684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:09:24.192764 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:24.192725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:09:24.193205 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:24.193119    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:09:24.193296 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:09:24.193276    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:09:37.169568 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:37.169532    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:09:37.169987 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:09:37.169719    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:09:49.170548 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:09:49.170511    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:09:49.170945 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:09:49.170692    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:10:00.178019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:00.176700    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:10:00.178019 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:10:00.177000    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:10:12.170346 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:12.170309    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:10:12.170777 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:10:12.170498    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:10:23.169966 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:23.169880    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:10:23.170462 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:10:23.170126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:10:34.170385 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:34.170347    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:10:34.170849 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:10:34.170534    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:10:45.170402 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:45.170362    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:10:45.928697 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:45.928669    2563 generic.go:356] "Generic (PLEG): container finished" podID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerID="8ce84a9e8760236c27d7ab19dad442e40851539baaa8252ac76cea531dffa6a5" exitCode=1
Jul 18 01:10:45.928829 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:45.928714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerDied","Data":"8ce84a9e8760236c27d7ab19dad442e40851539baaa8252ac76cea531dffa6a5"}
Jul 18 01:10:45.928829 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:45.928764    2563 scope.go:117] "RemoveContainer" containerID="70f6bdef93101efb40a8c09a615b2dff9e44b57182d1552a0a7110372a32b53d"
Jul 18 01:10:45.929128 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:45.929110    2563 scope.go:117] "RemoveContainer" containerID="8ce84a9e8760236c27d7ab19dad442e40851539baaa8252ac76cea531dffa6a5"
Jul 18 01:10:45.929355 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:10:45.929335    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:10:49.920887 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:49.920850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:10:49.921285 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:49.921216    2563 scope.go:117] "RemoveContainer" containerID="8ce84a9e8760236c27d7ab19dad442e40851539baaa8252ac76cea531dffa6a5"
Jul 18 01:10:49.921420 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:10:49.921397    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:10:51.135145 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:51.135106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:10:54.192613 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:54.192574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:10:54.193168 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:54.193021    2563 scope.go:117] "RemoveContainer" containerID="8ce84a9e8760236c27d7ab19dad442e40851539baaa8252ac76cea531dffa6a5"
Jul 18 01:10:54.193296 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:10:54.193273    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-9bd91509-77f8999676-h4z7z_odh-ai-gateway-infra(52274670-4bc0-49fe-b98b-f220a2860e5e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e"
Jul 18 01:10:54.266625 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:54.266591    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"]
Jul 18 01:10:55.117859 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.117828    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:10:55.168308 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.168269    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52274670-4bc0-49fe-b98b-f220a2860e5e-openshift-service-ca\") pod \"52274670-4bc0-49fe-b98b-f220a2860e5e\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") "
Jul 18 01:10:55.168476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.168346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls\") pod \"52274670-4bc0-49fe-b98b-f220a2860e5e\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") "
Jul 18 01:10:55.168515 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.168472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrvzl\" (UniqueName: \"kubernetes.io/projected/52274670-4bc0-49fe-b98b-f220a2860e5e-kube-api-access-hrvzl\") pod \"52274670-4bc0-49fe-b98b-f220a2860e5e\" (UID: \"52274670-4bc0-49fe-b98b-f220a2860e5e\") "
Jul 18 01:10:55.168725 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.168694    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52274670-4bc0-49fe-b98b-f220a2860e5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52274670-4bc0-49fe-b98b-f220a2860e5e" (UID: "52274670-4bc0-49fe-b98b-f220a2860e5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:10:55.170530 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.170505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52274670-4bc0-49fe-b98b-f220a2860e5e-kube-api-access-hrvzl" (OuterVolumeSpecName: "kube-api-access-hrvzl") pod "52274670-4bc0-49fe-b98b-f220a2860e5e" (UID: "52274670-4bc0-49fe-b98b-f220a2860e5e"). InnerVolumeSpecName "kube-api-access-hrvzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:10:55.170632 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.170596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52274670-4bc0-49fe-b98b-f220a2860e5e" (UID: "52274670-4bc0-49fe-b98b-f220a2860e5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:10:55.269939 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.269839    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52274670-4bc0-49fe-b98b-f220a2860e5e-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:10:55.269939 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.269875    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52274670-4bc0-49fe-b98b-f220a2860e5e-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:10:55.269939 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.269885    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrvzl\" (UniqueName: \"kubernetes.io/projected/52274670-4bc0-49fe-b98b-f220a2860e5e-kube-api-access-hrvzl\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:10:55.985596 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.985558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z" event={"ID":"52274670-4bc0-49fe-b98b-f220a2860e5e","Type":"ContainerDied","Data":"e896883e482b6c7c396c48657dd29274ecbb1673d6ec838072943868dd58f998"}
Jul 18 01:10:55.985596 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.985590    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"
Jul 18 01:10:55.985880 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:55.985629    2563 scope.go:117] "RemoveContainer" containerID="8ce84a9e8760236c27d7ab19dad442e40851539baaa8252ac76cea531dffa6a5"
Jul 18 01:10:56.009072 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:56.009034    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"]
Jul 18 01:10:56.013628 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:56.013600    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd91509-77f8999676-h4z7z"]
Jul 18 01:10:56.176076 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:10:56.176022    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" path="/var/lib/kubelet/pods/52274670-4bc0-49fe-b98b-f220a2860e5e/volumes"
Jul 18 01:11:01.945463 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.945352    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"]
Jul 18 01:11:01.946008 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.945984    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946012    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946038    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946046    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946083    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946092    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946107    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946115    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946124    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946131    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946140    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946147    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946251    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946264    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946274    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.946383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.946286    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:11:01.950772 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.950750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:01.953342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.953319    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bf4f6c8d-serving-cert\""
Jul 18 01:11:01.961475 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:01.961442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"]
Jul 18 01:11:02.025826 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.025784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba853dd-35c1-4a44-a663-59cf52fab5b2-openshift-service-ca\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.026001 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.025892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba853dd-35c1-4a44-a663-59cf52fab5b2-maas-api-tls\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.026001 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.025967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tr5b\" (UniqueName: \"kubernetes.io/projected/0ba853dd-35c1-4a44-a663-59cf52fab5b2-kube-api-access-2tr5b\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.127068 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.127007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tr5b\" (UniqueName: \"kubernetes.io/projected/0ba853dd-35c1-4a44-a663-59cf52fab5b2-kube-api-access-2tr5b\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.127289 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.127091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba853dd-35c1-4a44-a663-59cf52fab5b2-openshift-service-ca\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.127289 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.127166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba853dd-35c1-4a44-a663-59cf52fab5b2-maas-api-tls\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.127774 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.127749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba853dd-35c1-4a44-a663-59cf52fab5b2-openshift-service-ca\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.129631 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.129607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba853dd-35c1-4a44-a663-59cf52fab5b2-maas-api-tls\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.135277 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.135255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tr5b\" (UniqueName: \"kubernetes.io/projected/0ba853dd-35c1-4a44-a663-59cf52fab5b2-kube-api-access-2tr5b\") pod \"maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.264193 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.264094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:02.419762 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:02.419735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"]
Jul 18 01:11:02.421401 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:11:02.421373    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba853dd_35c1_4a44_a663_59cf52fab5b2.slice/crio-cac9071ca57d7958d0da8d3f824d6b854704dce873d08c388e42204dd7a703b0 WatchSource:0}: Error finding container cac9071ca57d7958d0da8d3f824d6b854704dce873d08c388e42204dd7a703b0: Status 404 returned error can't find the container with id cac9071ca57d7958d0da8d3f824d6b854704dce873d08c388e42204dd7a703b0
Jul 18 01:11:03.028415 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:03.028383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" event={"ID":"0ba853dd-35c1-4a44-a663-59cf52fab5b2","Type":"ContainerStarted","Data":"3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813"}
Jul 18 01:11:03.028741 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:03.028440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" event={"ID":"0ba853dd-35c1-4a44-a663-59cf52fab5b2","Type":"ContainerStarted","Data":"cac9071ca57d7958d0da8d3f824d6b854704dce873d08c388e42204dd7a703b0"}
Jul 18 01:11:03.028741 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:03.028609    2563 scope.go:117] "RemoveContainer" containerID="3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813"
Jul 18 01:11:04.036032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.035995    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerID="3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813" exitCode=1
Jul 18 01:11:04.036032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.036021    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerID="d8e1230149d825aa14523d0082c305ae3d0c861cd20c4e53bafa171873c48e00" exitCode=1
Jul 18 01:11:04.036498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.036087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" event={"ID":"0ba853dd-35c1-4a44-a663-59cf52fab5b2","Type":"ContainerDied","Data":"3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813"}
Jul 18 01:11:04.036498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.036134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" event={"ID":"0ba853dd-35c1-4a44-a663-59cf52fab5b2","Type":"ContainerDied","Data":"d8e1230149d825aa14523d0082c305ae3d0c861cd20c4e53bafa171873c48e00"}
Jul 18 01:11:04.036498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.036158    2563 scope.go:117] "RemoveContainer" containerID="3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813"
Jul 18 01:11:04.036498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.036433    2563 scope.go:117] "RemoveContainer" containerID="d8e1230149d825aa14523d0082c305ae3d0c861cd20c4e53bafa171873c48e00"
Jul 18 01:11:04.036699 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:04.036624    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:11:04.047704 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.047681    2563 scope.go:117] "RemoveContainer" containerID="3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813"
Jul 18 01:11:04.047969 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:04.047944    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813\": container with ID starting with 3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813 not found: ID does not exist" containerID="3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813"
Jul 18 01:11:04.048018 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.047978    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813"} err="failed to get container status \"3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813\": rpc error: code = NotFound desc = could not find container \"3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813\": container with ID starting with 3ed55ee7460a1d37b2c31e57433cd27c9c9eb4f36b950b1225ab33e313d15813 not found: ID does not exist"
Jul 18 01:11:04.756627 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:04.756591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:11:05.049185 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:05.049102    2563 scope.go:117] "RemoveContainer" containerID="d8e1230149d825aa14523d0082c305ae3d0c861cd20c4e53bafa171873c48e00"
Jul 18 01:11:05.049530 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:05.049278    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:11:07.264662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:07.264625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:07.265215 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:07.265195    2563 scope.go:117] "RemoveContainer" containerID="d8e1230149d825aa14523d0082c305ae3d0c861cd20c4e53bafa171873c48e00"
Jul 18 01:11:07.265444 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:07.265421    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:11:12.264425 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:12.264377    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:12.266834 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:12.264948    2563 scope.go:117] "RemoveContainer" containerID="d8e1230149d825aa14523d0082c305ae3d0c861cd20c4e53bafa171873c48e00"
Jul 18 01:11:12.266834 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:12.265172    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:11:23.169611 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:23.169575    2563 scope.go:117] "RemoveContainer" containerID="d8e1230149d825aa14523d0082c305ae3d0c861cd20c4e53bafa171873c48e00"
Jul 18 01:11:24.155413 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:24.155321    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerID="a1d63697a83bb3135dd644d567a845a178f5e0c0baeadc1d253456aebdfd35c1" exitCode=1
Jul 18 01:11:24.155413 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:24.155390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" event={"ID":"0ba853dd-35c1-4a44-a663-59cf52fab5b2","Type":"ContainerDied","Data":"a1d63697a83bb3135dd644d567a845a178f5e0c0baeadc1d253456aebdfd35c1"}
Jul 18 01:11:24.155658 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:24.155452    2563 scope.go:117] "RemoveContainer" containerID="d8e1230149d825aa14523d0082c305ae3d0c861cd20c4e53bafa171873c48e00"
Jul 18 01:11:24.155816 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:24.155800    2563 scope.go:117] "RemoveContainer" containerID="a1d63697a83bb3135dd644d567a845a178f5e0c0baeadc1d253456aebdfd35c1"
Jul 18 01:11:24.156028 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:24.156008    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:11:27.265138 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:27.265098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:27.265556 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:27.265421    2563 scope.go:117] "RemoveContainer" containerID="a1d63697a83bb3135dd644d567a845a178f5e0c0baeadc1d253456aebdfd35c1"
Jul 18 01:11:27.265613 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:27.265594    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:11:32.265181 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:32.265131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:32.265719 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:32.265646    2563 scope.go:117] "RemoveContainer" containerID="a1d63697a83bb3135dd644d567a845a178f5e0c0baeadc1d253456aebdfd35c1"
Jul 18 01:11:32.265896 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:32.265872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:11:46.170468 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:46.170432    2563 scope.go:117] "RemoveContainer" containerID="a1d63697a83bb3135dd644d567a845a178f5e0c0baeadc1d253456aebdfd35c1"
Jul 18 01:11:47.294618 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:47.294578    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerID="7b5a813fe2c133922e1cda200d39317c2ccf38604e148ff916fd00dab6d174bb" exitCode=1
Jul 18 01:11:47.295076 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:47.294631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" event={"ID":"0ba853dd-35c1-4a44-a663-59cf52fab5b2","Type":"ContainerDied","Data":"7b5a813fe2c133922e1cda200d39317c2ccf38604e148ff916fd00dab6d174bb"}
Jul 18 01:11:47.295076 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:47.294712    2563 scope.go:117] "RemoveContainer" containerID="a1d63697a83bb3135dd644d567a845a178f5e0c0baeadc1d253456aebdfd35c1"
Jul 18 01:11:47.295076 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:47.295023    2563 scope.go:117] "RemoveContainer" containerID="7b5a813fe2c133922e1cda200d39317c2ccf38604e148ff916fd00dab6d174bb"
Jul 18 01:11:47.295297 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:47.295257    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:11:52.264969 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:52.264936    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:52.265373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:52.264990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:11:52.265373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:11:52.265309    2563 scope.go:117] "RemoveContainer" containerID="7b5a813fe2c133922e1cda200d39317c2ccf38604e148ff916fd00dab6d174bb"
Jul 18 01:11:52.265490 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:11:52.265470    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:12:06.170286 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:06.170257    2563 scope.go:117] "RemoveContainer" containerID="7b5a813fe2c133922e1cda200d39317c2ccf38604e148ff916fd00dab6d174bb"
Jul 18 01:12:06.170801 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:06.170450    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk_odh-ai-gateway-infra(0ba853dd-35c1-4a44-a663-59cf52fab5b2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2"
Jul 18 01:12:09.334853 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:09.334811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:12:11.769993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:11.769952    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"]
Jul 18 01:12:11.930189 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:11.930125    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:12:12.033273 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.033169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba853dd-35c1-4a44-a663-59cf52fab5b2-openshift-service-ca\") pod \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") "
Jul 18 01:12:12.033273 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.033282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tr5b\" (UniqueName: \"kubernetes.io/projected/0ba853dd-35c1-4a44-a663-59cf52fab5b2-kube-api-access-2tr5b\") pod \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") "
Jul 18 01:12:12.033476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.033317    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba853dd-35c1-4a44-a663-59cf52fab5b2-maas-api-tls\") pod \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\" (UID: \"0ba853dd-35c1-4a44-a663-59cf52fab5b2\") "
Jul 18 01:12:12.033542 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.033513    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba853dd-35c1-4a44-a663-59cf52fab5b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ba853dd-35c1-4a44-a663-59cf52fab5b2" (UID: "0ba853dd-35c1-4a44-a663-59cf52fab5b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:12:12.035455 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.035425    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba853dd-35c1-4a44-a663-59cf52fab5b2-kube-api-access-2tr5b" (OuterVolumeSpecName: "kube-api-access-2tr5b") pod "0ba853dd-35c1-4a44-a663-59cf52fab5b2" (UID: "0ba853dd-35c1-4a44-a663-59cf52fab5b2"). InnerVolumeSpecName "kube-api-access-2tr5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:12:12.035455 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.035428    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba853dd-35c1-4a44-a663-59cf52fab5b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ba853dd-35c1-4a44-a663-59cf52fab5b2" (UID: "0ba853dd-35c1-4a44-a663-59cf52fab5b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:12:12.133936 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.133902    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba853dd-35c1-4a44-a663-59cf52fab5b2-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:12:12.133936 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.133929    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tr5b\" (UniqueName: \"kubernetes.io/projected/0ba853dd-35c1-4a44-a663-59cf52fab5b2-kube-api-access-2tr5b\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:12:12.133936 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.133940    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba853dd-35c1-4a44-a663-59cf52fab5b2-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:12:12.439560 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.439474    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"
Jul 18 01:12:12.439560 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.439510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk" event={"ID":"0ba853dd-35c1-4a44-a663-59cf52fab5b2","Type":"ContainerDied","Data":"cac9071ca57d7958d0da8d3f824d6b854704dce873d08c388e42204dd7a703b0"}
Jul 18 01:12:12.439749 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.439590    2563 scope.go:117] "RemoveContainer" containerID="7b5a813fe2c133922e1cda200d39317c2ccf38604e148ff916fd00dab6d174bb"
Jul 18 01:12:12.455739 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.455712    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"]
Jul 18 01:12:12.459117 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:12.459090    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bf4f6c8d-7b4f958969-mt2rk"]
Jul 18 01:12:14.176474 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:14.176439    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" path="/var/lib/kubelet/pods/0ba853dd-35c1-4a44-a663-59cf52fab5b2/volumes"
Jul 18 01:12:19.414689 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.414649    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"]
Jul 18 01:12:19.415219 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415203    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:12:19.415272 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415223    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:12:19.415272 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:12:19.415272 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415243    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:12:19.415363 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415341    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:12:19.415363 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415352    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:12:19.415430 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415362    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:12:19.415430 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415370    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:12:19.415430 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.415382    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52274670-4bc0-49fe-b98b-f220a2860e5e" containerName="maas-api"
Jul 18 01:12:19.420086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.420047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:19.422749 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.422720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4701d191-serving-cert\""
Jul 18 01:12:19.426579 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.426544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"]
Jul 18 01:12:19.501280 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.501239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46gl\" (UniqueName: \"kubernetes.io/projected/c7e6a391-d319-4764-a9a2-32808d8a2b9b-kube-api-access-j46gl\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:19.501490 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.501310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e6a391-d319-4764-a9a2-32808d8a2b9b-openshift-service-ca\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:19.501490 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.501377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:19.602286 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.602243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e6a391-d319-4764-a9a2-32808d8a2b9b-openshift-service-ca\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:19.602501 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.602325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:19.602501 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.602362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j46gl\" (UniqueName: \"kubernetes.io/projected/c7e6a391-d319-4764-a9a2-32808d8a2b9b-kube-api-access-j46gl\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:19.602501 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:19.602481    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-serving-cert: secret "maas-api-e2e-ait-4701d191-serving-cert" not found
Jul 18 01:12:19.602665 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:19.602562    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls podName:c7e6a391-d319-4764-a9a2-32808d8a2b9b nodeName:}" failed. No retries permitted until 2026-07-18 01:12:20.102539471 +0000 UTC m=+1910.422516846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls") pod "maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" (UID: "c7e6a391-d319-4764-a9a2-32808d8a2b9b") : secret "maas-api-e2e-ait-4701d191-serving-cert" not found
Jul 18 01:12:19.603033 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.603009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e6a391-d319-4764-a9a2-32808d8a2b9b-openshift-service-ca\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:19.624266 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:19.624236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46gl\" (UniqueName: \"kubernetes.io/projected/c7e6a391-d319-4764-a9a2-32808d8a2b9b-kube-api-access-j46gl\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:20.107433 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:20.107395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:20.110087 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:20.110047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls\") pod \"maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:20.333986 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:20.333944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:20.480118 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:20.480080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"]
Jul 18 01:12:20.487099 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:12:20.487032    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e6a391_d319_4764_a9a2_32808d8a2b9b.slice/crio-0db1066cd54bf1aaadc7e2439e4601c718e46a57cc68a9dd627879416ebb06a7 WatchSource:0}: Error finding container 0db1066cd54bf1aaadc7e2439e4601c718e46a57cc68a9dd627879416ebb06a7: Status 404 returned error can't find the container with id 0db1066cd54bf1aaadc7e2439e4601c718e46a57cc68a9dd627879416ebb06a7
Jul 18 01:12:21.495496 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:21.495455    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerID="01de2485e3e70a0c729ee6fd43cc15e414da663e2f318600393e9791005f90bb" exitCode=1
Jul 18 01:12:21.495870 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:21.495541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerDied","Data":"01de2485e3e70a0c729ee6fd43cc15e414da663e2f318600393e9791005f90bb"}
Jul 18 01:12:21.495870 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:21.495593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerStarted","Data":"0db1066cd54bf1aaadc7e2439e4601c718e46a57cc68a9dd627879416ebb06a7"}
Jul 18 01:12:21.495870 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:21.495730    2563 scope.go:117] "RemoveContainer" containerID="01de2485e3e70a0c729ee6fd43cc15e414da663e2f318600393e9791005f90bb"
Jul 18 01:12:22.247788 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:22.247746    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:12:22.503437 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:22.503345    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerID="eac235c6530a8d2e9ad858208bae6ec3e79b210ea15cb2cb0048133d5a48f9ab" exitCode=1
Jul 18 01:12:22.503437 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:22.503404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerDied","Data":"eac235c6530a8d2e9ad858208bae6ec3e79b210ea15cb2cb0048133d5a48f9ab"}
Jul 18 01:12:22.503885 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:22.503465    2563 scope.go:117] "RemoveContainer" containerID="01de2485e3e70a0c729ee6fd43cc15e414da663e2f318600393e9791005f90bb"
Jul 18 01:12:22.503885 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:22.503708    2563 scope.go:117] "RemoveContainer" containerID="eac235c6530a8d2e9ad858208bae6ec3e79b210ea15cb2cb0048133d5a48f9ab"
Jul 18 01:12:22.504002 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:22.503932    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:12:23.511186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:23.511160    2563 scope.go:117] "RemoveContainer" containerID="eac235c6530a8d2e9ad858208bae6ec3e79b210ea15cb2cb0048133d5a48f9ab"
Jul 18 01:12:23.511562 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:23.511337    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:12:25.334414 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:25.334372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:25.334824 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:25.334809    2563 scope.go:117] "RemoveContainer" containerID="eac235c6530a8d2e9ad858208bae6ec3e79b210ea15cb2cb0048133d5a48f9ab"
Jul 18 01:12:25.335000 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:25.334978    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:12:30.334759 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:30.334715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:30.335327 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:30.335292    2563 scope.go:117] "RemoveContainer" containerID="eac235c6530a8d2e9ad858208bae6ec3e79b210ea15cb2cb0048133d5a48f9ab"
Jul 18 01:12:30.335500 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:30.335481    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:12:45.169966 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:45.169923    2563 scope.go:117] "RemoveContainer" containerID="eac235c6530a8d2e9ad858208bae6ec3e79b210ea15cb2cb0048133d5a48f9ab"
Jul 18 01:12:45.170483 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:45.170291    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 01:12:46.644205 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:46.644169    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerID="dbe5500fb81512cd55fd6c7ceca293fa907539597ee6999a5a58a09bf42ce7ff" exitCode=1
Jul 18 01:12:46.644661 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:46.644218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerDied","Data":"dbe5500fb81512cd55fd6c7ceca293fa907539597ee6999a5a58a09bf42ce7ff"}
Jul 18 01:12:46.644661 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:46.644268    2563 scope.go:117] "RemoveContainer" containerID="eac235c6530a8d2e9ad858208bae6ec3e79b210ea15cb2cb0048133d5a48f9ab"
Jul 18 01:12:46.644661 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:46.644542    2563 scope.go:117] "RemoveContainer" containerID="dbe5500fb81512cd55fd6c7ceca293fa907539597ee6999a5a58a09bf42ce7ff"
Jul 18 01:12:46.644782 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:46.644755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:12:50.334568 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:50.334528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:50.334982 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:50.334598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:12:50.334982 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:12:50.334933    2563 scope.go:117] "RemoveContainer" containerID="dbe5500fb81512cd55fd6c7ceca293fa907539597ee6999a5a58a09bf42ce7ff"
Jul 18 01:12:50.335154 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:12:50.335134    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:13:04.178411 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:04.178382    2563 scope.go:117] "RemoveContainer" containerID="dbe5500fb81512cd55fd6c7ceca293fa907539597ee6999a5a58a09bf42ce7ff"
Jul 18 01:13:04.178792 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:13:04.178569    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:13:18.170165 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:18.170123    2563 scope.go:117] "RemoveContainer" containerID="dbe5500fb81512cd55fd6c7ceca293fa907539597ee6999a5a58a09bf42ce7ff"
Jul 18 01:13:18.834414 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:18.834376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerStarted","Data":"41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"}
Jul 18 01:13:18.834623 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:18.834599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:13:18.850768 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:18.850700    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podStartSLOduration=58.130761159 podStartE2EDuration="59.850678635s" totalImagesPullingTime="1.719917476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:12:19 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:13:18.849900397 +0000 UTC m=+1969.169877783" watchObservedRunningTime="2026-07-18 01:13:18.850678635 +0000 UTC m=+1969.170656019"
Jul 18 01:13:19.841490 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:19.841454    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerID="41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be" exitCode=1
Jul 18 01:13:19.841926 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:19.841532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerDied","Data":"41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"}
Jul 18 01:13:19.841926 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:19.841598    2563 scope.go:117] "RemoveContainer" containerID="dbe5500fb81512cd55fd6c7ceca293fa907539597ee6999a5a58a09bf42ce7ff"
Jul 18 01:13:19.841926 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:19.841849    2563 scope.go:117] "RemoveContainer" containerID="41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"
Jul 18 01:13:19.842099 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:13:19.842032    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:13:20.334931 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:20.334887    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:13:20.848693 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:20.848661    2563 scope.go:117] "RemoveContainer" containerID="41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"
Jul 18 01:13:20.849083 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:13:20.848881    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:13:21.853800 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:21.853766    2563 scope.go:117] "RemoveContainer" containerID="41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"
Jul 18 01:13:21.854223 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:13:21.853952    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:13:36.170173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:36.170140    2563 scope.go:117] "RemoveContainer" containerID="41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"
Jul 18 01:13:36.170576 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:13:36.170357    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:13:51.170727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:13:51.170690    2563 scope.go:117] "RemoveContainer" containerID="41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"
Jul 18 01:13:51.171249 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:13:51.170875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:14:06.170475 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:06.170439    2563 scope.go:117] "RemoveContainer" containerID="41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"
Jul 18 01:14:07.103270 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:07.103239    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df" exitCode=1
Jul 18 01:14:07.103438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:07.103306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerDied","Data":"1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"}
Jul 18 01:14:07.103438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:07.103370    2563 scope.go:117] "RemoveContainer" containerID="41b3fee9f9536a195d48744904c4c25b6edafa8eb7f1d80d435384174cd7f0be"
Jul 18 01:14:07.103709 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:07.103690    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:14:07.103897 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:14:07.103875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:14:08.834780 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:08.834743    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:14:08.835228 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:08.835116    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:14:08.835328 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:14:08.835306    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:14:10.334194 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:10.334158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:14:10.334596 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:10.334560    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:14:10.334757 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:14:10.334736    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:14:22.170452 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:22.170415    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:14:22.170943 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:14:22.170624    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:14:34.170089 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:34.170029    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:14:34.170574 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:14:34.170262    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:14:48.169833 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:14:48.169792    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:14:48.170314 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:14:48.170039    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:15:00.173567 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:00.173535    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:15:00.174091 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:15:00.173799    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:15:13.169744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:13.169700    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:15:13.170175 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:15:13.169873    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:15:25.170139 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:25.170108    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:15:25.170545 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:15:25.170325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:15:40.172513 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:40.172475    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:15:41.619476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:41.619440    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302" exitCode=1
Jul 18 01:15:41.619954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:41.619511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerDied","Data":"32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"}
Jul 18 01:15:41.619954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:41.619573    2563 scope.go:117] "RemoveContainer" containerID="1640e4eaeac8baf897fba9a09c16a8b3b2e722b87b56580a3b7fcbf0d61523df"
Jul 18 01:15:41.620101 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:41.619959    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:15:41.620239 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:15:41.620216    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:15:43.834883 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:43.834843    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:15:43.835318 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:43.835260    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:15:43.835466 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:15:43.835443    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:15:50.335186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:50.335146    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:15:50.335592 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:15:50.335566    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:15:50.335770 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:15:50.335750    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:16:05.170165 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:16:05.170131    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:16:05.170549 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:16:05.170324    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:16:20.180240 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:16:20.180209    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:16:20.182706 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:16:20.180399    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:16:34.169986 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:16:34.169899    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:16:34.170410 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:16:34.170115    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:16:45.170492 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:16:45.170451    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:16:45.170960 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:16:45.170659    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:16:57.169622 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:16:57.169577    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:16:57.170121 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:16:57.169786    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:17:10.173767 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:17:10.173734    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:17:10.174318 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:17:10.173942    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:17:23.169848 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:17:23.169808    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:17:23.170377 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:17:23.169994    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:17:35.170188 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:17:35.170152    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:17:35.170746 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:17:35.170358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:17:46.170348 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:17:46.170314    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:17:46.170746 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:17:46.170512    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:17:57.170044 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:17:57.170001    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:17:57.170477 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:17:57.170229    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:18:09.170506 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:09.170461    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:18:09.171091 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:09.170706    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:18:23.170495 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:23.170454    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:18:23.170998 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:23.170758    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 01:18:24.528901 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:24.528869    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerID="b209c39b7abbecdb314631feca676ef3ce7d897fb2f165014d51e21ff0cf9b13" exitCode=1
Jul 18 01:18:24.529323 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:24.528934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerDied","Data":"b209c39b7abbecdb314631feca676ef3ce7d897fb2f165014d51e21ff0cf9b13"}
Jul 18 01:18:24.529323 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:24.528985    2563 scope.go:117] "RemoveContainer" containerID="32588f337b04fd00299301fc8e6f98459e329211d3250d7001aeeefe05942302"
Jul 18 01:18:24.529438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:24.529356    2563 scope.go:117] "RemoveContainer" containerID="b209c39b7abbecdb314631feca676ef3ce7d897fb2f165014d51e21ff0cf9b13"
Jul 18 01:18:24.529578 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:24.529556    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:18:27.533203 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:27.533165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:18:28.835223 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:28.835180    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:18:28.835626 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:28.835556    2563 scope.go:117] "RemoveContainer" containerID="b209c39b7abbecdb314631feca676ef3ce7d897fb2f165014d51e21ff0cf9b13"
Jul 18 01:18:28.835756 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:28.835734    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph_odh-ai-gateway-infra(c7e6a391-d319-4764-a9a2-32808d8a2b9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b"
Jul 18 01:18:29.936759 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:29.936721    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"]
Jul 18 01:18:30.098806 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.098776    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:18:30.170051 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.170015    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j46gl\" (UniqueName: \"kubernetes.io/projected/c7e6a391-d319-4764-a9a2-32808d8a2b9b-kube-api-access-j46gl\") pod \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") "
Jul 18 01:18:30.170262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.170078    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls\") pod \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") "
Jul 18 01:18:30.170262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.170182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e6a391-d319-4764-a9a2-32808d8a2b9b-openshift-service-ca\") pod \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\" (UID: \"c7e6a391-d319-4764-a9a2-32808d8a2b9b\") "
Jul 18 01:18:30.170612 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.170575    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e6a391-d319-4764-a9a2-32808d8a2b9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7e6a391-d319-4764-a9a2-32808d8a2b9b" (UID: "c7e6a391-d319-4764-a9a2-32808d8a2b9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:18:30.172275 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.172237    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e6a391-d319-4764-a9a2-32808d8a2b9b-kube-api-access-j46gl" (OuterVolumeSpecName: "kube-api-access-j46gl") pod "c7e6a391-d319-4764-a9a2-32808d8a2b9b" (UID: "c7e6a391-d319-4764-a9a2-32808d8a2b9b"). InnerVolumeSpecName "kube-api-access-j46gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:18:30.172461 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.172442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7e6a391-d319-4764-a9a2-32808d8a2b9b" (UID: "c7e6a391-d319-4764-a9a2-32808d8a2b9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:18:30.271782 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.271731    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e6a391-d319-4764-a9a2-32808d8a2b9b-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.271782 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.271769    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46gl\" (UniqueName: \"kubernetes.io/projected/c7e6a391-d319-4764-a9a2-32808d8a2b9b-kube-api-access-j46gl\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.272105 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.271848    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e6a391-d319-4764-a9a2-32808d8a2b9b-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:18:30.563455 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.563427    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"
Jul 18 01:18:30.563639 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.563457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph" event={"ID":"c7e6a391-d319-4764-a9a2-32808d8a2b9b","Type":"ContainerDied","Data":"0db1066cd54bf1aaadc7e2439e4601c718e46a57cc68a9dd627879416ebb06a7"}
Jul 18 01:18:30.563639 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.563527    2563 scope.go:117] "RemoveContainer" containerID="b209c39b7abbecdb314631feca676ef3ce7d897fb2f165014d51e21ff0cf9b13"
Jul 18 01:18:30.581583 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.581550    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"]
Jul 18 01:18:30.589230 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:30.589198    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4701d191-84f5fd5dbb-6nfph"]
Jul 18 01:18:32.176404 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:32.176370    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" path="/var/lib/kubelet/pods/c7e6a391-d319-4764-a9a2-32808d8a2b9b/volumes"
Jul 18 01:18:37.431846 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.431806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"]
Jul 18 01:18:37.432424 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432403    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432426    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432437    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432445    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432460    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432469    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432490    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432498    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432519    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432529    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:18:37.432534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:18:37.433103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432548    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.433103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.433103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432678    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.433103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432691    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.433103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432700    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.433103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432709    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba853dd-35c1-4a44-a663-59cf52fab5b2" containerName="maas-api"
Jul 18 01:18:37.433103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432720    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.433103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.432729    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:18:37.436231 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.436178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:37.439197 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.439172    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4e26d7cc-serving-cert\""
Jul 18 01:18:37.447629 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.447563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"]
Jul 18 01:18:37.544130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.544079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:37.544130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.544136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897605bf-5721-40f4-b5cd-5b738d0f7bed-openshift-service-ca\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:37.544377 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.544263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsb8\" (UniqueName: \"kubernetes.io/projected/897605bf-5721-40f4-b5cd-5b738d0f7bed-kube-api-access-nhsb8\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:37.645142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.645095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:37.645142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.645140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897605bf-5721-40f4-b5cd-5b738d0f7bed-openshift-service-ca\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:37.645412 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.645204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsb8\" (UniqueName: \"kubernetes.io/projected/897605bf-5721-40f4-b5cd-5b738d0f7bed-kube-api-access-nhsb8\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:37.645412 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:37.645260    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-serving-cert: secret "maas-api-e2e-derive-4e26d7cc-serving-cert" not found
Jul 18 01:18:37.645412 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:37.645378    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls podName:897605bf-5721-40f4-b5cd-5b738d0f7bed nodeName:}" failed. No retries permitted until 2026-07-18 01:18:38.145352479 +0000 UTC m=+2288.465329855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls") pod "maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" (UID: "897605bf-5721-40f4-b5cd-5b738d0f7bed") : secret "maas-api-e2e-derive-4e26d7cc-serving-cert" not found
Jul 18 01:18:37.646012 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.645987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897605bf-5721-40f4-b5cd-5b738d0f7bed-openshift-service-ca\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:37.667389 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:37.667354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsb8\" (UniqueName: \"kubernetes.io/projected/897605bf-5721-40f4-b5cd-5b738d0f7bed-kube-api-access-nhsb8\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:38.150872 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:38.150824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:38.153439 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:38.153411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls\") pod \"maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:38.349983 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:38.349943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:38.501398 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:38.501329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"]
Jul 18 01:18:38.504403 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:18:38.504359    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897605bf_5721_40f4_b5cd_5b738d0f7bed.slice/crio-dc1e6e1b02b24c5f5a32a668ed6538ffe88663b23d00e340d4dd7c1e7288fed6 WatchSource:0}: Error finding container dc1e6e1b02b24c5f5a32a668ed6538ffe88663b23d00e340d4dd7c1e7288fed6: Status 404 returned error can't find the container with id dc1e6e1b02b24c5f5a32a668ed6538ffe88663b23d00e340d4dd7c1e7288fed6
Jul 18 01:18:38.610154 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:38.610113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerStarted","Data":"dc1e6e1b02b24c5f5a32a668ed6538ffe88663b23d00e340d4dd7c1e7288fed6"}
Jul 18 01:18:39.617465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:39.617369    2563 generic.go:356] "Generic (PLEG): container finished" podID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerID="a45991c4c6f768972380f0b871a96ed456c8fd24e1fbee6ac22b692486689c77" exitCode=1
Jul 18 01:18:39.617465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:39.617438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerDied","Data":"a45991c4c6f768972380f0b871a96ed456c8fd24e1fbee6ac22b692486689c77"}
Jul 18 01:18:39.617866 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:39.617668    2563 scope.go:117] "RemoveContainer" containerID="a45991c4c6f768972380f0b871a96ed456c8fd24e1fbee6ac22b692486689c77"
Jul 18 01:18:40.624519 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:40.624428    2563 generic.go:356] "Generic (PLEG): container finished" podID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerID="6120e3588ea301ec3c466b17510ce138063219cb95b768f20df71c2d7861bdd8" exitCode=1
Jul 18 01:18:40.624917 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:40.624514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerDied","Data":"6120e3588ea301ec3c466b17510ce138063219cb95b768f20df71c2d7861bdd8"}
Jul 18 01:18:40.624917 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:40.624585    2563 scope.go:117] "RemoveContainer" containerID="a45991c4c6f768972380f0b871a96ed456c8fd24e1fbee6ac22b692486689c77"
Jul 18 01:18:40.624917 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:40.624854    2563 scope.go:117] "RemoveContainer" containerID="6120e3588ea301ec3c466b17510ce138063219cb95b768f20df71c2d7861bdd8"
Jul 18 01:18:40.625106 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:40.625045    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:18:41.631824 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:41.631798    2563 scope.go:117] "RemoveContainer" containerID="6120e3588ea301ec3c466b17510ce138063219cb95b768f20df71c2d7861bdd8"
Jul 18 01:18:41.632205 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:41.632016    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:18:43.350520 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:43.350484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:43.350914 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:43.350900    2563 scope.go:117] "RemoveContainer" containerID="6120e3588ea301ec3c466b17510ce138063219cb95b768f20df71c2d7861bdd8"
Jul 18 01:18:43.351115 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:43.351085    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:18:45.230581 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:45.230546    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:18:48.351088 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:48.351032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:18:48.351488 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:18:48.351455    2563 scope.go:117] "RemoveContainer" containerID="6120e3588ea301ec3c466b17510ce138063219cb95b768f20df71c2d7861bdd8"
Jul 18 01:18:48.351639 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:18:48.351619    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:19:00.172474 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:00.172441    2563 scope.go:117] "RemoveContainer" containerID="6120e3588ea301ec3c466b17510ce138063219cb95b768f20df71c2d7861bdd8"
Jul 18 01:19:00.738216 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:00.738182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerStarted","Data":"0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967"}
Jul 18 01:19:00.738407 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:00.738390    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:19:00.757508 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:00.757393    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podStartSLOduration=22.741849604 podStartE2EDuration="23.757373633s" totalImagesPullingTime="1.015524029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:18:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:19:00.754929043 +0000 UTC m=+2311.074906427" watchObservedRunningTime="2026-07-18 01:19:00.757373633 +0000 UTC m=+2311.077351055"
Jul 18 01:19:01.745240 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:01.745206    2563 generic.go:356] "Generic (PLEG): container finished" podID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerID="0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967" exitCode=1
Jul 18 01:19:01.745642 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:01.745296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerDied","Data":"0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967"}
Jul 18 01:19:01.745642 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:01.745356    2563 scope.go:117] "RemoveContainer" containerID="6120e3588ea301ec3c466b17510ce138063219cb95b768f20df71c2d7861bdd8"
Jul 18 01:19:01.745642 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:01.745503    2563 scope.go:117] "RemoveContainer" containerID="0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967"
Jul 18 01:19:01.745804 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:19:01.745718    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:19:02.752485 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:02.752447    2563 scope.go:117] "RemoveContainer" containerID="0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967"
Jul 18 01:19:02.752897 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:19:02.752658    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:19:08.350943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:08.350880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:19:08.351405 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:08.351364    2563 scope.go:117] "RemoveContainer" containerID="0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967"
Jul 18 01:19:08.351571 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:19:08.351549    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:19:19.170340 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:19.170304    2563 scope.go:117] "RemoveContainer" containerID="0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967"
Jul 18 01:19:19.170748 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:19:19.170459    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:19:33.169914 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:33.169826    2563 scope.go:117] "RemoveContainer" containerID="0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967"
Jul 18 01:19:33.927258 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:33.927225    2563 generic.go:356] "Generic (PLEG): container finished" podID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerID="8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a" exitCode=1
Jul 18 01:19:33.927410 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:33.927294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerDied","Data":"8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a"}
Jul 18 01:19:33.927410 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:33.927355    2563 scope.go:117] "RemoveContainer" containerID="0502271d5b2b84a54f8aebaae5350d683030e5ee7f8d57b2ded01760a908f967"
Jul 18 01:19:33.927675 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:33.927658    2563 scope.go:117] "RemoveContainer" containerID="8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a"
Jul 18 01:19:33.927855 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:19:33.927833    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:19:35.739123 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:35.739041    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:19:35.739545 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:35.739515    2563 scope.go:117] "RemoveContainer" containerID="8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a"
Jul 18 01:19:35.739752 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:19:35.739731    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:19:38.350286 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:38.350244    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:19:38.350686 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:38.350675    2563 scope.go:117] "RemoveContainer" containerID="8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a"
Jul 18 01:19:38.350858 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:19:38.350837    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:19:50.173756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:19:50.173721    2563 scope.go:117] "RemoveContainer" containerID="8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a"
Jul 18 01:19:50.174182 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:19:50.173958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:20:02.169974 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:02.169940    2563 scope.go:117] "RemoveContainer" containerID="8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a"
Jul 18 01:20:02.170428 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:20:02.170179    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:20:14.170381 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:14.170342    2563 scope.go:117] "RemoveContainer" containerID="8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a"
Jul 18 01:20:15.157413 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:15.157323    2563 generic.go:356] "Generic (PLEG): container finished" podID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea" exitCode=1
Jul 18 01:20:15.157413 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:15.157396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerDied","Data":"7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"}
Jul 18 01:20:15.157595 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:15.157496    2563 scope.go:117] "RemoveContainer" containerID="8572ab1bbef021b2dd69c566bb3922458e08b7dc447b033913f1f911308c7b9a"
Jul 18 01:20:15.157916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:15.157898    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:20:15.158152 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:20:15.158130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:20:15.739446 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:15.739410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:20:16.164831 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:16.164746    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:20:16.164983 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:20:16.164925    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:20:18.351045 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:18.351005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:20:18.351449 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:18.351401    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:20:18.351593 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:20:18.351572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:20:32.170104 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:32.170045    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:20:32.170528 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:20:32.170303    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:20:47.169511 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:47.169467    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:20:47.170192 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:20:47.169705    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:20:58.170606 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:20:58.170561    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:20:58.171112 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:20:58.170791    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:21:13.169797 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:13.169760    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:21:13.170226 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:21:13.169943    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:21:26.169948 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:26.169912    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:21:26.172424 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:21:26.170159    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:21:37.169514 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:37.169481    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:21:38.611672 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:38.611634    2563 generic.go:356] "Generic (PLEG): container finished" podID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerID="e4fb5a0cde4e20e666d37569125306c32f214685ee444658029842970106f7eb" exitCode=1
Jul 18 01:21:38.612180 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:38.611704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerDied","Data":"e4fb5a0cde4e20e666d37569125306c32f214685ee444658029842970106f7eb"}
Jul 18 01:21:38.612180 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:38.611769    2563 scope.go:117] "RemoveContainer" containerID="7175616c825961001d23376c16a0430408706e78e0ab2a127c5aee320b4f45ea"
Jul 18 01:21:38.612180 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:38.612145    2563 scope.go:117] "RemoveContainer" containerID="e4fb5a0cde4e20e666d37569125306c32f214685ee444658029842970106f7eb"
Jul 18 01:21:38.612406 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:21:38.612383    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:21:40.739090 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:40.739040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:21:40.739577 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:40.739534    2563 scope.go:117] "RemoveContainer" containerID="e4fb5a0cde4e20e666d37569125306c32f214685ee444658029842970106f7eb"
Jul 18 01:21:40.739771 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:21:40.739746    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns_odh-ai-gateway-infra(897605bf-5721-40f4-b5cd-5b738d0f7bed)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed"
Jul 18 01:21:44.635119 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:44.635072    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:21:47.137789 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.137753    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"]
Jul 18 01:21:47.287740 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.287703    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:21:47.392347 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.392274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhsb8\" (UniqueName: \"kubernetes.io/projected/897605bf-5721-40f4-b5cd-5b738d0f7bed-kube-api-access-nhsb8\") pod \"897605bf-5721-40f4-b5cd-5b738d0f7bed\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") "
Jul 18 01:21:47.392347 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.392311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897605bf-5721-40f4-b5cd-5b738d0f7bed-openshift-service-ca\") pod \"897605bf-5721-40f4-b5cd-5b738d0f7bed\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") "
Jul 18 01:21:47.392347 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.392346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls\") pod \"897605bf-5721-40f4-b5cd-5b738d0f7bed\" (UID: \"897605bf-5721-40f4-b5cd-5b738d0f7bed\") "
Jul 18 01:21:47.392720 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.392694    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897605bf-5721-40f4-b5cd-5b738d0f7bed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "897605bf-5721-40f4-b5cd-5b738d0f7bed" (UID: "897605bf-5721-40f4-b5cd-5b738d0f7bed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:21:47.394366 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.394339    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897605bf-5721-40f4-b5cd-5b738d0f7bed-kube-api-access-nhsb8" (OuterVolumeSpecName: "kube-api-access-nhsb8") pod "897605bf-5721-40f4-b5cd-5b738d0f7bed" (UID: "897605bf-5721-40f4-b5cd-5b738d0f7bed"). InnerVolumeSpecName "kube-api-access-nhsb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:21:47.394536 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.394513    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "897605bf-5721-40f4-b5cd-5b738d0f7bed" (UID: "897605bf-5721-40f4-b5cd-5b738d0f7bed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:21:47.493577 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.493540    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsb8\" (UniqueName: \"kubernetes.io/projected/897605bf-5721-40f4-b5cd-5b738d0f7bed-kube-api-access-nhsb8\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.493577 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.493571    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897605bf-5721-40f4-b5cd-5b738d0f7bed-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.493577 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.493581    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897605bf-5721-40f4-b5cd-5b738d0f7bed-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:21:47.667007 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.666925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"
Jul 18 01:21:47.667007 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.666965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns" event={"ID":"897605bf-5721-40f4-b5cd-5b738d0f7bed","Type":"ContainerDied","Data":"dc1e6e1b02b24c5f5a32a668ed6538ffe88663b23d00e340d4dd7c1e7288fed6"}
Jul 18 01:21:47.667226 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.667028    2563 scope.go:117] "RemoveContainer" containerID="e4fb5a0cde4e20e666d37569125306c32f214685ee444658029842970106f7eb"
Jul 18 01:21:47.687721 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.687688    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"]
Jul 18 01:21:47.692250 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:47.692224    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4e26d7cc-77d896d6b6-l68ns"]
Jul 18 01:21:48.175940 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:48.175909    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" path="/var/lib/kubelet/pods/897605bf-5721-40f4-b5cd-5b738d0f7bed/volumes"
Jul 18 01:21:58.851254 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851208    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"]
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851590    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851602    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851613    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851619    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851636    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851651    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851656    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851664    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851671    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851679    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851684    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851739    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851747    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851753    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851759    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851765    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:21:58.851836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.851771    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:21:58.856142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.856121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"
Jul 18 01:21:58.858582 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.858561    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:21:58.862293 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.862268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"]
Jul 18 01:21:58.993114 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:58.993046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b608b341-6686-4424-8e94-ce67f501aff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k\" (UID: \"b608b341-6686-4424-8e94-ce67f501aff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"
Jul 18 01:21:59.094473 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:59.094434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b608b341-6686-4424-8e94-ce67f501aff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k\" (UID: \"b608b341-6686-4424-8e94-ce67f501aff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"
Jul 18 01:21:59.095195 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:59.095170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b608b341-6686-4424-8e94-ce67f501aff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k\" (UID: \"b608b341-6686-4424-8e94-ce67f501aff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"
Jul 18 01:21:59.168464 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:59.168368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"
Jul 18 01:21:59.513460 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:59.513427    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"]
Jul 18 01:21:59.515131 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:21:59.515100    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb608b341_6686_4424_8e94_ce67f501aff0.slice/crio-d134e63bca128678809d0ee28c68c92262570fc2d0e8835a84f6d42822be5890 WatchSource:0}: Error finding container d134e63bca128678809d0ee28c68c92262570fc2d0e8835a84f6d42822be5890: Status 404 returned error can't find the container with id d134e63bca128678809d0ee28c68c92262570fc2d0e8835a84f6d42822be5890
Jul 18 01:21:59.735068 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:21:59.735018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k" event={"ID":"b608b341-6686-4424-8e94-ce67f501aff0","Type":"ContainerStarted","Data":"d134e63bca128678809d0ee28c68c92262570fc2d0e8835a84f6d42822be5890"}
Jul 18 01:22:00.742359 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:00.742324    2563 generic.go:356] "Generic (PLEG): container finished" podID="b608b341-6686-4424-8e94-ce67f501aff0" containerID="ad2e196858b26fbfa929a6547a5e597cdebec8076d4c989dcb540f56a4547430" exitCode=6
Jul 18 01:22:00.742763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:00.742410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k" event={"ID":"b608b341-6686-4424-8e94-ce67f501aff0","Type":"ContainerDied","Data":"ad2e196858b26fbfa929a6547a5e597cdebec8076d4c989dcb540f56a4547430"}
Jul 18 01:22:00.742763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:00.742599    2563 scope.go:117] "RemoveContainer" containerID="ad2e196858b26fbfa929a6547a5e597cdebec8076d4c989dcb540f56a4547430"
Jul 18 01:22:01.749734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:01.749643    2563 generic.go:356] "Generic (PLEG): container finished" podID="b608b341-6686-4424-8e94-ce67f501aff0" containerID="451e0a3bca35a61219f0170687f9e1226479fc947127a26c4002bcf74f2c89fd" exitCode=6
Jul 18 01:22:01.750220 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:01.749732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k" event={"ID":"b608b341-6686-4424-8e94-ce67f501aff0","Type":"ContainerDied","Data":"451e0a3bca35a61219f0170687f9e1226479fc947127a26c4002bcf74f2c89fd"}
Jul 18 01:22:01.750220 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:01.749801    2563 scope.go:117] "RemoveContainer" containerID="ad2e196858b26fbfa929a6547a5e597cdebec8076d4c989dcb540f56a4547430"
Jul 18 01:22:01.750220 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:01.750130    2563 scope.go:117] "RemoveContainer" containerID="451e0a3bca35a61219f0170687f9e1226479fc947127a26c4002bcf74f2c89fd"
Jul 18 01:22:01.750380 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:22:01.750354    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k_odh-ai-gateway-infra(b608b341-6686-4424-8e94-ce67f501aff0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k" podUID="b608b341-6686-4424-8e94-ce67f501aff0"
Jul 18 01:22:02.757007 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:02.756978    2563 scope.go:117] "RemoveContainer" containerID="451e0a3bca35a61219f0170687f9e1226479fc947127a26c4002bcf74f2c89fd"
Jul 18 01:22:02.757419 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:22:02.757199    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k_odh-ai-gateway-infra(b608b341-6686-4424-8e94-ce67f501aff0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k" podUID="b608b341-6686-4424-8e94-ce67f501aff0"
Jul 18 01:22:16.170531 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:16.170487    2563 scope.go:117] "RemoveContainer" containerID="451e0a3bca35a61219f0170687f9e1226479fc947127a26c4002bcf74f2c89fd"
Jul 18 01:22:16.837128 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:16.837094    2563 generic.go:356] "Generic (PLEG): container finished" podID="b608b341-6686-4424-8e94-ce67f501aff0" containerID="6a51646172a45baaa8a096900415e7ed529c200cdce2c7d025d253876817a4ca" exitCode=6
Jul 18 01:22:16.837288 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:16.837171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k" event={"ID":"b608b341-6686-4424-8e94-ce67f501aff0","Type":"ContainerDied","Data":"6a51646172a45baaa8a096900415e7ed529c200cdce2c7d025d253876817a4ca"}
Jul 18 01:22:16.837288 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:16.837241    2563 scope.go:117] "RemoveContainer" containerID="451e0a3bca35a61219f0170687f9e1226479fc947127a26c4002bcf74f2c89fd"
Jul 18 01:22:16.837583 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:16.837564    2563 scope.go:117] "RemoveContainer" containerID="6a51646172a45baaa8a096900415e7ed529c200cdce2c7d025d253876817a4ca"
Jul 18 01:22:16.837829 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:22:16.837810    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k_odh-ai-gateway-infra(b608b341-6686-4424-8e94-ce67f501aff0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k" podUID="b608b341-6686-4424-8e94-ce67f501aff0"
Jul 18 01:22:17.863365 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:17.863328    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"]
Jul 18 01:22:18.003894 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.003866    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"
Jul 18 01:22:18.166732 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.166638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b608b341-6686-4424-8e94-ce67f501aff0-maas-api-service-ca\") pod \"b608b341-6686-4424-8e94-ce67f501aff0\" (UID: \"b608b341-6686-4424-8e94-ce67f501aff0\") "
Jul 18 01:22:18.167032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.167009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b608b341-6686-4424-8e94-ce67f501aff0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b608b341-6686-4424-8e94-ce67f501aff0" (UID: "b608b341-6686-4424-8e94-ce67f501aff0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:22:18.267843 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.267810    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b608b341-6686-4424-8e94-ce67f501aff0-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:22:18.850570 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.850530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k" event={"ID":"b608b341-6686-4424-8e94-ce67f501aff0","Type":"ContainerDied","Data":"d134e63bca128678809d0ee28c68c92262570fc2d0e8835a84f6d42822be5890"}
Jul 18 01:22:18.850570 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.850558    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"
Jul 18 01:22:18.850826 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.850604    2563 scope.go:117] "RemoveContainer" containerID="6a51646172a45baaa8a096900415e7ed529c200cdce2c7d025d253876817a4ca"
Jul 18 01:22:18.867029 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.866998    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"]
Jul 18 01:22:18.872925 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:18.872900    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-g6n5k"]
Jul 18 01:22:20.176063 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:22:20.176017    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b608b341-6686-4424-8e94-ce67f501aff0" path="/var/lib/kubelet/pods/b608b341-6686-4424-8e94-ce67f501aff0/volumes"
Jul 18 01:23:14.947577 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947541    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"]
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947930    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947942    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947947    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947958    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947964    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947971    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.947976    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e6a391-d319-4764-a9a2-32808d8a2b9b" containerName="maas-api"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.948035    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.948044    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.948051    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:14.948098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.948087    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:14.951259 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.951239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"
Jul 18 01:23:14.953621 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.953593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:23:14.957002 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:14.956971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"]
Jul 18 01:23:15.048165 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:15.048127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a18fa98-6e76-4146-9100-d731d898b871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-z4qjx\" (UID: \"0a18fa98-6e76-4146-9100-d731d898b871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"
Jul 18 01:23:15.149438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:15.149393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a18fa98-6e76-4146-9100-d731d898b871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-z4qjx\" (UID: \"0a18fa98-6e76-4146-9100-d731d898b871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"
Jul 18 01:23:15.150047 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:15.150023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a18fa98-6e76-4146-9100-d731d898b871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-z4qjx\" (UID: \"0a18fa98-6e76-4146-9100-d731d898b871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"
Jul 18 01:23:15.263375 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:15.263284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"
Jul 18 01:23:15.341420 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:15.341384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:23:15.406839 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:15.406813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"]
Jul 18 01:23:15.408723 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:23:15.408694    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a18fa98_6e76_4146_9100_d731d898b871.slice/crio-3b9de1dfb16db4451954c442dbb63596b630bebb6a43fc28f7d6740d1ddb1bfc WatchSource:0}: Error finding container 3b9de1dfb16db4451954c442dbb63596b630bebb6a43fc28f7d6740d1ddb1bfc: Status 404 returned error can't find the container with id 3b9de1dfb16db4451954c442dbb63596b630bebb6a43fc28f7d6740d1ddb1bfc
Jul 18 01:23:16.166272 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:16.166233    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a18fa98-6e76-4146-9100-d731d898b871" containerID="198295ba09c6084cfc2253ae4b4e977fe0f46f1a5c92e5d9ad1283f2cd685031" exitCode=6
Jul 18 01:23:16.166667 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:16.166281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx" event={"ID":"0a18fa98-6e76-4146-9100-d731d898b871","Type":"ContainerDied","Data":"198295ba09c6084cfc2253ae4b4e977fe0f46f1a5c92e5d9ad1283f2cd685031"}
Jul 18 01:23:16.166667 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:16.166373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx" event={"ID":"0a18fa98-6e76-4146-9100-d731d898b871","Type":"ContainerStarted","Data":"3b9de1dfb16db4451954c442dbb63596b630bebb6a43fc28f7d6740d1ddb1bfc"}
Jul 18 01:23:16.166667 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:16.166582    2563 scope.go:117] "RemoveContainer" containerID="198295ba09c6084cfc2253ae4b4e977fe0f46f1a5c92e5d9ad1283f2cd685031"
Jul 18 01:23:17.173610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:17.173526    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a18fa98-6e76-4146-9100-d731d898b871" containerID="9e132ad2fd6a0a1cc2646477b2e6a9100299a1a928593531b3981d113329db98" exitCode=6
Jul 18 01:23:17.173610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:17.173576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx" event={"ID":"0a18fa98-6e76-4146-9100-d731d898b871","Type":"ContainerDied","Data":"9e132ad2fd6a0a1cc2646477b2e6a9100299a1a928593531b3981d113329db98"}
Jul 18 01:23:17.174035 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:17.173622    2563 scope.go:117] "RemoveContainer" containerID="198295ba09c6084cfc2253ae4b4e977fe0f46f1a5c92e5d9ad1283f2cd685031"
Jul 18 01:23:17.174035 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:17.173800    2563 scope.go:117] "RemoveContainer" containerID="9e132ad2fd6a0a1cc2646477b2e6a9100299a1a928593531b3981d113329db98"
Jul 18 01:23:17.174169 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:17.174080    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-z4qjx_odh-ai-gateway-infra(0a18fa98-6e76-4146-9100-d731d898b871)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx" podUID="0a18fa98-6e76-4146-9100-d731d898b871"
Jul 18 01:23:18.179934 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:18.179902    2563 scope.go:117] "RemoveContainer" containerID="9e132ad2fd6a0a1cc2646477b2e6a9100299a1a928593531b3981d113329db98"
Jul 18 01:23:18.180347 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:18.180112    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-z4qjx_odh-ai-gateway-infra(0a18fa98-6e76-4146-9100-d731d898b871)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx" podUID="0a18fa98-6e76-4146-9100-d731d898b871"
Jul 18 01:23:22.050570 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.050531    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"]
Jul 18 01:23:22.051029 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.050937    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:22.051029 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.050948    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b608b341-6686-4424-8e94-ce67f501aff0" containerName="revoke-keys"
Jul 18 01:23:22.051123 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.051046    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="897605bf-5721-40f4-b5cd-5b738d0f7bed" containerName="maas-api"
Jul 18 01:23:22.055380 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.055361    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"
Jul 18 01:23:22.061221 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.061186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"]
Jul 18 01:23:22.100628 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.100585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115a4d2-ed4c-4c21-b1e6-4b88f843901a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h\" (UID: \"2115a4d2-ed4c-4c21-b1e6-4b88f843901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"
Jul 18 01:23:22.201810 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.201771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115a4d2-ed4c-4c21-b1e6-4b88f843901a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h\" (UID: \"2115a4d2-ed4c-4c21-b1e6-4b88f843901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"
Jul 18 01:23:22.202467 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.202442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115a4d2-ed4c-4c21-b1e6-4b88f843901a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h\" (UID: \"2115a4d2-ed4c-4c21-b1e6-4b88f843901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"
Jul 18 01:23:22.367232 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.367138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"
Jul 18 01:23:22.503889 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:22.503859    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"]
Jul 18 01:23:22.505664 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:23:22.505636    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2115a4d2_ed4c_4c21_b1e6_4b88f843901a.slice/crio-f90dfdc72f5e74606f5657d77ab07d181799c39e0c582032a75c69b137f134fc WatchSource:0}: Error finding container f90dfdc72f5e74606f5657d77ab07d181799c39e0c582032a75c69b137f134fc: Status 404 returned error can't find the container with id f90dfdc72f5e74606f5657d77ab07d181799c39e0c582032a75c69b137f134fc
Jul 18 01:23:23.211583 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:23.211554    2563 generic.go:356] "Generic (PLEG): container finished" podID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerID="54e3c4d36a28d897254186970744b0220829bacee8910ae240e0e042f0dc6031" exitCode=6
Jul 18 01:23:23.211907 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:23.211599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h" event={"ID":"2115a4d2-ed4c-4c21-b1e6-4b88f843901a","Type":"ContainerDied","Data":"54e3c4d36a28d897254186970744b0220829bacee8910ae240e0e042f0dc6031"}
Jul 18 01:23:23.211907 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:23.211666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h" event={"ID":"2115a4d2-ed4c-4c21-b1e6-4b88f843901a","Type":"ContainerStarted","Data":"f90dfdc72f5e74606f5657d77ab07d181799c39e0c582032a75c69b137f134fc"}
Jul 18 01:23:23.211907 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:23.211896    2563 scope.go:117] "RemoveContainer" containerID="54e3c4d36a28d897254186970744b0220829bacee8910ae240e0e042f0dc6031"
Jul 18 01:23:24.218626 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:24.218586    2563 generic.go:356] "Generic (PLEG): container finished" podID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerID="6b9ed3300a7b303084e246e8328349245c06fde525fbdebeae5fe3b82cc17e5b" exitCode=6
Jul 18 01:23:24.219045 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:24.218649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h" event={"ID":"2115a4d2-ed4c-4c21-b1e6-4b88f843901a","Type":"ContainerDied","Data":"6b9ed3300a7b303084e246e8328349245c06fde525fbdebeae5fe3b82cc17e5b"}
Jul 18 01:23:24.219045 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:24.218710    2563 scope.go:117] "RemoveContainer" containerID="54e3c4d36a28d897254186970744b0220829bacee8910ae240e0e042f0dc6031"
Jul 18 01:23:24.219045 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:24.218890    2563 scope.go:117] "RemoveContainer" containerID="6b9ed3300a7b303084e246e8328349245c06fde525fbdebeae5fe3b82cc17e5b"
Jul 18 01:23:24.219191 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:24.219160    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h_odh-ai-gateway-infra(2115a4d2-ed4c-4c21-b1e6-4b88f843901a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a"
Jul 18 01:23:24.432414 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:24.432370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:23:25.225712 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:25.225678    2563 scope.go:117] "RemoveContainer" containerID="6b9ed3300a7b303084e246e8328349245c06fde525fbdebeae5fe3b82cc17e5b"
Jul 18 01:23:25.226199 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:25.225952    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h_odh-ai-gateway-infra(2115a4d2-ed4c-4c21-b1e6-4b88f843901a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a"
Jul 18 01:23:26.942536 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:26.942498    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"]
Jul 18 01:23:26.947405 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:26.947383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"
Jul 18 01:23:26.952081 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:26.952034    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"]
Jul 18 01:23:27.044283 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:27.044241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faf7e80-4bf2-4a62-adef-d7872837ce71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv\" (UID: \"9faf7e80-4bf2-4a62-adef-d7872837ce71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"
Jul 18 01:23:27.145582 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:27.145541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faf7e80-4bf2-4a62-adef-d7872837ce71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv\" (UID: \"9faf7e80-4bf2-4a62-adef-d7872837ce71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"
Jul 18 01:23:27.146246 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:27.146223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faf7e80-4bf2-4a62-adef-d7872837ce71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv\" (UID: \"9faf7e80-4bf2-4a62-adef-d7872837ce71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"
Jul 18 01:23:27.259605 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:27.259571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"
Jul 18 01:23:27.397667 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:27.397632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"]
Jul 18 01:23:27.399153 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:23:27.399117    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9faf7e80_4bf2_4a62_adef_d7872837ce71.slice/crio-e49619c88d5b182368d009a7b4f3527c6a58e3ebc0be7d51386b2d5d8238bf86 WatchSource:0}: Error finding container e49619c88d5b182368d009a7b4f3527c6a58e3ebc0be7d51386b2d5d8238bf86: Status 404 returned error can't find the container with id e49619c88d5b182368d009a7b4f3527c6a58e3ebc0be7d51386b2d5d8238bf86
Jul 18 01:23:28.243860 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:28.243819    2563 generic.go:356] "Generic (PLEG): container finished" podID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerID="ace6266e8a6d74d173505ed2c32f741f1873899186be9dda9412e3f875f40d05" exitCode=6
Jul 18 01:23:28.244296 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:28.243860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv" event={"ID":"9faf7e80-4bf2-4a62-adef-d7872837ce71","Type":"ContainerDied","Data":"ace6266e8a6d74d173505ed2c32f741f1873899186be9dda9412e3f875f40d05"}
Jul 18 01:23:28.244296 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:28.243909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv" event={"ID":"9faf7e80-4bf2-4a62-adef-d7872837ce71","Type":"ContainerStarted","Data":"e49619c88d5b182368d009a7b4f3527c6a58e3ebc0be7d51386b2d5d8238bf86"}
Jul 18 01:23:28.244296 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:28.244109    2563 scope.go:117] "RemoveContainer" containerID="ace6266e8a6d74d173505ed2c32f741f1873899186be9dda9412e3f875f40d05"
Jul 18 01:23:29.251684 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:29.251651    2563 generic.go:356] "Generic (PLEG): container finished" podID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerID="8f0ea341c00cb04ecade19134a56680fd57ef3671431ac2f0fd164fd4ed4cbd9" exitCode=6
Jul 18 01:23:29.252188 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:29.251741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv" event={"ID":"9faf7e80-4bf2-4a62-adef-d7872837ce71","Type":"ContainerDied","Data":"8f0ea341c00cb04ecade19134a56680fd57ef3671431ac2f0fd164fd4ed4cbd9"}
Jul 18 01:23:29.252188 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:29.251822    2563 scope.go:117] "RemoveContainer" containerID="ace6266e8a6d74d173505ed2c32f741f1873899186be9dda9412e3f875f40d05"
Jul 18 01:23:29.252188 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:29.252086    2563 scope.go:117] "RemoveContainer" containerID="8f0ea341c00cb04ecade19134a56680fd57ef3671431ac2f0fd164fd4ed4cbd9"
Jul 18 01:23:29.252360 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:29.252294    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv_odh-ai-gateway-infra(9faf7e80-4bf2-4a62-adef-d7872837ce71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71"
Jul 18 01:23:30.267643 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:30.267612    2563 scope.go:117] "RemoveContainer" containerID="8f0ea341c00cb04ecade19134a56680fd57ef3671431ac2f0fd164fd4ed4cbd9"
Jul 18 01:23:30.268021 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:30.267806    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv_odh-ai-gateway-infra(9faf7e80-4bf2-4a62-adef-d7872837ce71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71"
Jul 18 01:23:32.170434 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.170398    2563 scope.go:117] "RemoveContainer" containerID="9e132ad2fd6a0a1cc2646477b2e6a9100299a1a928593531b3981d113329db98"
Jul 18 01:23:32.482420 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.482383    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"]
Jul 18 01:23:32.486715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.486684    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:32.489505 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.489473    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d1e63f47-serving-cert\""
Jul 18 01:23:32.497681 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.497653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"]
Jul 18 01:23:32.597385 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.597357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpkd\" (UniqueName: \"kubernetes.io/projected/5393597a-6bd5-48bf-94a8-cce3509aa936-kube-api-access-qhpkd\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:32.597513 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.597405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:32.597513 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.597470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5393597a-6bd5-48bf-94a8-cce3509aa936-openshift-service-ca\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:32.698476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.698443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5393597a-6bd5-48bf-94a8-cce3509aa936-openshift-service-ca\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:32.698670 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.698527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpkd\" (UniqueName: \"kubernetes.io/projected/5393597a-6bd5-48bf-94a8-cce3509aa936-kube-api-access-qhpkd\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:32.698670 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.698554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:32.698670 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:32.698651    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-serving-cert: secret "maas-api-e2e-mt-d1e63f47-serving-cert" not found
Jul 18 01:23:32.698834 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:32.698704    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls podName:5393597a-6bd5-48bf-94a8-cce3509aa936 nodeName:}" failed. No retries permitted until 2026-07-18 01:23:33.198688593 +0000 UTC m=+2583.518665954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls") pod "maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8" (UID: "5393597a-6bd5-48bf-94a8-cce3509aa936") : secret "maas-api-e2e-mt-d1e63f47-serving-cert" not found
Jul 18 01:23:32.699211 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.699190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5393597a-6bd5-48bf-94a8-cce3509aa936-openshift-service-ca\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:32.710528 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:32.710494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpkd\" (UniqueName: \"kubernetes.io/projected/5393597a-6bd5-48bf-94a8-cce3509aa936-kube-api-access-qhpkd\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:33.203351 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:33.203315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:33.203848 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:33.203460    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-serving-cert: secret "maas-api-e2e-mt-d1e63f47-serving-cert" not found
Jul 18 01:23:33.203848 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:33.203533    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls podName:5393597a-6bd5-48bf-94a8-cce3509aa936 nodeName:}" failed. No retries permitted until 2026-07-18 01:23:34.203510884 +0000 UTC m=+2584.523488271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls") pod "maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8" (UID: "5393597a-6bd5-48bf-94a8-cce3509aa936") : secret "maas-api-e2e-mt-d1e63f47-serving-cert" not found
Jul 18 01:23:33.287522 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:33.287486    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a18fa98-6e76-4146-9100-d731d898b871" containerID="0ae2130000dfb88ebe113788609ad047aec85d6ef6fe6020741da27620b86c82" exitCode=6
Jul 18 01:23:33.287676 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:33.287558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx" event={"ID":"0a18fa98-6e76-4146-9100-d731d898b871","Type":"ContainerDied","Data":"0ae2130000dfb88ebe113788609ad047aec85d6ef6fe6020741da27620b86c82"}
Jul 18 01:23:33.287676 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:33.287633    2563 scope.go:117] "RemoveContainer" containerID="9e132ad2fd6a0a1cc2646477b2e6a9100299a1a928593531b3981d113329db98"
Jul 18 01:23:33.287947 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:33.287929    2563 scope.go:117] "RemoveContainer" containerID="0ae2130000dfb88ebe113788609ad047aec85d6ef6fe6020741da27620b86c82"
Jul 18 01:23:33.288211 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:33.288179    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-z4qjx_odh-ai-gateway-infra(0a18fa98-6e76-4146-9100-d731d898b871)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx" podUID="0a18fa98-6e76-4146-9100-d731d898b871"
Jul 18 01:23:34.213458 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.213414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:34.215964 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.215936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls\") pod \"maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:34.301587 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.301554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:34.317024 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.316990    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"]
Jul 18 01:23:34.464188 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.464160    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"]
Jul 18 01:23:34.465849 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:23:34.465808    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5393597a_6bd5_48bf_94a8_cce3509aa936.slice/crio-574612d83a52cb6bf8b24b940645a0ede55edb555ed8b83b0e827a8e330ae6e3 WatchSource:0}: Error finding container 574612d83a52cb6bf8b24b940645a0ede55edb555ed8b83b0e827a8e330ae6e3: Status 404 returned error can't find the container with id 574612d83a52cb6bf8b24b940645a0ede55edb555ed8b83b0e827a8e330ae6e3
Jul 18 01:23:34.467284 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.467265    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 01:23:34.484103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.484074    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"
Jul 18 01:23:34.516016 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.515984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a18fa98-6e76-4146-9100-d731d898b871-maas-api-service-ca\") pod \"0a18fa98-6e76-4146-9100-d731d898b871\" (UID: \"0a18fa98-6e76-4146-9100-d731d898b871\") "
Jul 18 01:23:34.516406 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.516363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a18fa98-6e76-4146-9100-d731d898b871-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a18fa98-6e76-4146-9100-d731d898b871" (UID: "0a18fa98-6e76-4146-9100-d731d898b871"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:23:34.617356 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:34.617310    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a18fa98-6e76-4146-9100-d731d898b871-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:23:35.304813 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.304720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8" event={"ID":"5393597a-6bd5-48bf-94a8-cce3509aa936","Type":"ContainerStarted","Data":"1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9"}
Jul 18 01:23:35.304813 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.304791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8" event={"ID":"5393597a-6bd5-48bf-94a8-cce3509aa936","Type":"ContainerStarted","Data":"574612d83a52cb6bf8b24b940645a0ede55edb555ed8b83b0e827a8e330ae6e3"}
Jul 18 01:23:35.305489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.304957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:35.306552 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.306520    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"
Jul 18 01:23:35.306680 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.306543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx" event={"ID":"0a18fa98-6e76-4146-9100-d731d898b871","Type":"ContainerDied","Data":"3b9de1dfb16db4451954c442dbb63596b630bebb6a43fc28f7d6740d1ddb1bfc"}
Jul 18 01:23:35.306680 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.306606    2563 scope.go:117] "RemoveContainer" containerID="0ae2130000dfb88ebe113788609ad047aec85d6ef6fe6020741da27620b86c82"
Jul 18 01:23:35.327177 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.327133    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8" podStartSLOduration=2.866124528 podStartE2EDuration="3.327115607s" totalImagesPullingTime="460.991079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:23:35.324089806 +0000 UTC m=+2585.644067189" watchObservedRunningTime="2026-07-18 01:23:35.327115607 +0000 UTC m=+2585.647092989"
Jul 18 01:23:35.338004 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.337975    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"]
Jul 18 01:23:35.341088 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:35.341039    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-z4qjx"]
Jul 18 01:23:36.170567 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:36.170532    2563 scope.go:117] "RemoveContainer" containerID="6b9ed3300a7b303084e246e8328349245c06fde525fbdebeae5fe3b82cc17e5b"
Jul 18 01:23:36.176519 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:36.176484    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a18fa98-6e76-4146-9100-d731d898b871" path="/var/lib/kubelet/pods/0a18fa98-6e76-4146-9100-d731d898b871/volumes"
Jul 18 01:23:37.320423 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:37.320335    2563 generic.go:356] "Generic (PLEG): container finished" podID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerID="27f6f34f7ee03df0f9297980ef061464b10fe3e1a5d93958629406dc3f6b7a95" exitCode=6
Jul 18 01:23:37.320423 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:37.320407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h" event={"ID":"2115a4d2-ed4c-4c21-b1e6-4b88f843901a","Type":"ContainerDied","Data":"27f6f34f7ee03df0f9297980ef061464b10fe3e1a5d93958629406dc3f6b7a95"}
Jul 18 01:23:37.320951 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:37.320470    2563 scope.go:117] "RemoveContainer" containerID="6b9ed3300a7b303084e246e8328349245c06fde525fbdebeae5fe3b82cc17e5b"
Jul 18 01:23:37.320951 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:37.320729    2563 scope.go:117] "RemoveContainer" containerID="27f6f34f7ee03df0f9297980ef061464b10fe3e1a5d93958629406dc3f6b7a95"
Jul 18 01:23:37.320951 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:37.320935    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h_odh-ai-gateway-infra(2115a4d2-ed4c-4c21-b1e6-4b88f843901a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a"
Jul 18 01:23:37.733483 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:37.733446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:23:38.346767 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:38.346727    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"]
Jul 18 01:23:38.488274 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:38.488247    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"
Jul 18 01:23:38.552982 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:38.552949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115a4d2-ed4c-4c21-b1e6-4b88f843901a-maas-api-service-ca\") pod \"2115a4d2-ed4c-4c21-b1e6-4b88f843901a\" (UID: \"2115a4d2-ed4c-4c21-b1e6-4b88f843901a\") "
Jul 18 01:23:38.553338 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:38.553315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2115a4d2-ed4c-4c21-b1e6-4b88f843901a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2115a4d2-ed4c-4c21-b1e6-4b88f843901a" (UID: "2115a4d2-ed4c-4c21-b1e6-4b88f843901a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:23:38.654689 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:38.654570    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2115a4d2-ed4c-4c21-b1e6-4b88f843901a-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:23:39.334864 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:39.334832    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"
Jul 18 01:23:39.335105 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:39.334889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h" event={"ID":"2115a4d2-ed4c-4c21-b1e6-4b88f843901a","Type":"ContainerDied","Data":"f90dfdc72f5e74606f5657d77ab07d181799c39e0c582032a75c69b137f134fc"}
Jul 18 01:23:39.335105 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:39.334957    2563 scope.go:117] "RemoveContainer" containerID="27f6f34f7ee03df0f9297980ef061464b10fe3e1a5d93958629406dc3f6b7a95"
Jul 18 01:23:39.359850 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:39.359822    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"]
Jul 18 01:23:39.365456 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:39.365428    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-bt54h"]
Jul 18 01:23:40.176812 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:40.176776    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" path="/var/lib/kubelet/pods/2115a4d2-ed4c-4c21-b1e6-4b88f843901a/volumes"
Jul 18 01:23:41.318833 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:41.318802    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:42.170491 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:42.170455    2563 scope.go:117] "RemoveContainer" containerID="8f0ea341c00cb04ecade19134a56680fd57ef3671431ac2f0fd164fd4ed4cbd9"
Jul 18 01:23:43.362107 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:43.362068    2563 generic.go:356] "Generic (PLEG): container finished" podID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerID="cf86c310b2201fc0f577dd1778ac051582e30586e99682c173148ec2785cbd5b" exitCode=6
Jul 18 01:23:43.362569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:43.362101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv" event={"ID":"9faf7e80-4bf2-4a62-adef-d7872837ce71","Type":"ContainerDied","Data":"cf86c310b2201fc0f577dd1778ac051582e30586e99682c173148ec2785cbd5b"}
Jul 18 01:23:43.362569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:43.362177    2563 scope.go:117] "RemoveContainer" containerID="8f0ea341c00cb04ecade19134a56680fd57ef3671431ac2f0fd164fd4ed4cbd9"
Jul 18 01:23:43.362569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:43.362534    2563 scope.go:117] "RemoveContainer" containerID="cf86c310b2201fc0f577dd1778ac051582e30586e99682c173148ec2785cbd5b"
Jul 18 01:23:43.362749 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:43.362729    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv_odh-ai-gateway-infra(9faf7e80-4bf2-4a62-adef-d7872837ce71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71"
Jul 18 01:23:44.398050 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:44.398011    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"]
Jul 18 01:23:44.542562 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:44.542534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"
Jul 18 01:23:44.607786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:44.607757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faf7e80-4bf2-4a62-adef-d7872837ce71-maas-api-service-ca\") pod \"9faf7e80-4bf2-4a62-adef-d7872837ce71\" (UID: \"9faf7e80-4bf2-4a62-adef-d7872837ce71\") "
Jul 18 01:23:44.608192 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:44.608170    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9faf7e80-4bf2-4a62-adef-d7872837ce71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9faf7e80-4bf2-4a62-adef-d7872837ce71" (UID: "9faf7e80-4bf2-4a62-adef-d7872837ce71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:23:44.709097 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:44.709051    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9faf7e80-4bf2-4a62-adef-d7872837ce71-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:23:45.376512 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:45.376479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"
Jul 18 01:23:45.376730 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:45.376513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv" event={"ID":"9faf7e80-4bf2-4a62-adef-d7872837ce71","Type":"ContainerDied","Data":"e49619c88d5b182368d009a7b4f3527c6a58e3ebc0be7d51386b2d5d8238bf86"}
Jul 18 01:23:45.376730 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:45.376591    2563 scope.go:117] "RemoveContainer" containerID="cf86c310b2201fc0f577dd1778ac051582e30586e99682c173148ec2785cbd5b"
Jul 18 01:23:45.397743 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:45.397709    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"]
Jul 18 01:23:45.401124 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:45.401095    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-v9nnv"]
Jul 18 01:23:46.175707 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:46.175672    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" path="/var/lib/kubelet/pods/9faf7e80-4bf2-4a62-adef-d7872837ce71/volumes"
Jul 18 01:23:54.845853 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:54.845808    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"]
Jul 18 01:23:54.846358 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:54.846096    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8" podUID="5393597a-6bd5-48bf-94a8-cce3509aa936" containerName="maas-api" containerID="cri-o://1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9" gracePeriod=30
Jul 18 01:23:55.103769 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.103708    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:55.201181 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.201140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhpkd\" (UniqueName: \"kubernetes.io/projected/5393597a-6bd5-48bf-94a8-cce3509aa936-kube-api-access-qhpkd\") pod \"5393597a-6bd5-48bf-94a8-cce3509aa936\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") "
Jul 18 01:23:55.201358 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.201202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5393597a-6bd5-48bf-94a8-cce3509aa936-openshift-service-ca\") pod \"5393597a-6bd5-48bf-94a8-cce3509aa936\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") "
Jul 18 01:23:55.201358 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.201244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls\") pod \"5393597a-6bd5-48bf-94a8-cce3509aa936\" (UID: \"5393597a-6bd5-48bf-94a8-cce3509aa936\") "
Jul 18 01:23:55.201612 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.201581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5393597a-6bd5-48bf-94a8-cce3509aa936-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5393597a-6bd5-48bf-94a8-cce3509aa936" (UID: "5393597a-6bd5-48bf-94a8-cce3509aa936"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:23:55.203331 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.203309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5393597a-6bd5-48bf-94a8-cce3509aa936-kube-api-access-qhpkd" (OuterVolumeSpecName: "kube-api-access-qhpkd") pod "5393597a-6bd5-48bf-94a8-cce3509aa936" (UID: "5393597a-6bd5-48bf-94a8-cce3509aa936"). InnerVolumeSpecName "kube-api-access-qhpkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:23:55.203420 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.203395    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5393597a-6bd5-48bf-94a8-cce3509aa936" (UID: "5393597a-6bd5-48bf-94a8-cce3509aa936"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:23:55.302477 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.302384    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhpkd\" (UniqueName: \"kubernetes.io/projected/5393597a-6bd5-48bf-94a8-cce3509aa936-kube-api-access-qhpkd\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:23:55.302477 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.302433    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5393597a-6bd5-48bf-94a8-cce3509aa936-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:23:55.302477 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.302458    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5393597a-6bd5-48bf-94a8-cce3509aa936-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:23:55.434610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.434509    2563 generic.go:356] "Generic (PLEG): container finished" podID="5393597a-6bd5-48bf-94a8-cce3509aa936" containerID="1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9" exitCode=0
Jul 18 01:23:55.434610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.434546    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"
Jul 18 01:23:55.434610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.434590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8" event={"ID":"5393597a-6bd5-48bf-94a8-cce3509aa936","Type":"ContainerDied","Data":"1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9"}
Jul 18 01:23:55.434881 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.434645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8" event={"ID":"5393597a-6bd5-48bf-94a8-cce3509aa936","Type":"ContainerDied","Data":"574612d83a52cb6bf8b24b940645a0ede55edb555ed8b83b0e827a8e330ae6e3"}
Jul 18 01:23:55.434881 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.434668    2563 scope.go:117] "RemoveContainer" containerID="1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9"
Jul 18 01:23:55.444221 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.444201    2563 scope.go:117] "RemoveContainer" containerID="1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9"
Jul 18 01:23:55.444477 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:23:55.444457    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9\": container with ID starting with 1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9 not found: ID does not exist" containerID="1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9"
Jul 18 01:23:55.444520 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.444486    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9"} err="failed to get container status \"1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9\": rpc error: code = NotFound desc = could not find container \"1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9\": container with ID starting with 1f5bfa701d0dd7d4495b6a8e2e4af8bcf7dd5ef4e8ba20dc098bce92b09f6de9 not found: ID does not exist"
Jul 18 01:23:55.456591 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.456549    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"]
Jul 18 01:23:55.459929 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:55.459884    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d1e63f47-759cdb5c68-wnhg8"]
Jul 18 01:23:56.176386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:56.176353    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5393597a-6bd5-48bf-94a8-cce3509aa936" path="/var/lib/kubelet/pods/5393597a-6bd5-48bf-94a8-cce3509aa936/volumes"
Jul 18 01:23:57.737364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:23:57.737328    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:24:12.090229 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090191    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"]
Jul 18 01:24:12.090606 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090592    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:12.090662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090607    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:12.090662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090616    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:12.090662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090622    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:12.090662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090629    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:12.090662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090635    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:12.090662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090642    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:12.090662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090648    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090669    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090674    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090681    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5393597a-6bd5-48bf-94a8-cce3509aa936" containerName="maas-api"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090686    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5393597a-6bd5-48bf-94a8-cce3509aa936" containerName="maas-api"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090692    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090697    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090752    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090760    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090766    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090771    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090779    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090783    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5393597a-6bd5-48bf-94a8-cce3509aa936" containerName="maas-api"
Jul 18 01:24:12.090874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.090789    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:12.093884 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.093865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"
Jul 18 01:24:12.096254 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.096231    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:24:12.104171 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.104143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"]
Jul 18 01:24:12.253921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.253880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ec4a9-12f7-4b33-ae3f-697fd5703bf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd\" (UID: \"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"
Jul 18 01:24:12.354945 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.354852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ec4a9-12f7-4b33-ae3f-697fd5703bf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd\" (UID: \"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"
Jul 18 01:24:12.355548 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.355529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ec4a9-12f7-4b33-ae3f-697fd5703bf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd\" (UID: \"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"
Jul 18 01:24:12.405604 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.405569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"
Jul 18 01:24:12.541706 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:12.541678    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"]
Jul 18 01:24:12.543532 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:24:12.543504    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ec4a9_12f7_4b33_ae3f_697fd5703bf3.slice/crio-21484af8361a2ffbcf19013834c3be344115a6f61761eca2f63582b6a91f80f9 WatchSource:0}: Error finding container 21484af8361a2ffbcf19013834c3be344115a6f61761eca2f63582b6a91f80f9: Status 404 returned error can't find the container with id 21484af8361a2ffbcf19013834c3be344115a6f61761eca2f63582b6a91f80f9
Jul 18 01:24:13.537739 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:13.537701    2563 generic.go:356] "Generic (PLEG): container finished" podID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerID="81b2fb092830810aa2f9f277bdf3707bcb24ee0f510f9428567221672e3cfed7" exitCode=6
Jul 18 01:24:13.538143 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:13.537787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" event={"ID":"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3","Type":"ContainerDied","Data":"81b2fb092830810aa2f9f277bdf3707bcb24ee0f510f9428567221672e3cfed7"}
Jul 18 01:24:13.538143 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:13.537846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" event={"ID":"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3","Type":"ContainerStarted","Data":"21484af8361a2ffbcf19013834c3be344115a6f61761eca2f63582b6a91f80f9"}
Jul 18 01:24:13.538143 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:13.538002    2563 scope.go:117] "RemoveContainer" containerID="81b2fb092830810aa2f9f277bdf3707bcb24ee0f510f9428567221672e3cfed7"
Jul 18 01:24:14.547834 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.547796    2563 generic.go:356] "Generic (PLEG): container finished" podID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerID="711d31a44871c2282e52ee5f1ab1e75a61977a6552e07487d1ec7dc1ae565ea1" exitCode=6
Jul 18 01:24:14.548297 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.547879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" event={"ID":"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3","Type":"ContainerDied","Data":"711d31a44871c2282e52ee5f1ab1e75a61977a6552e07487d1ec7dc1ae565ea1"}
Jul 18 01:24:14.548297 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.547942    2563 scope.go:117] "RemoveContainer" containerID="81b2fb092830810aa2f9f277bdf3707bcb24ee0f510f9428567221672e3cfed7"
Jul 18 01:24:14.548297 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.548117    2563 scope.go:117] "RemoveContainer" containerID="711d31a44871c2282e52ee5f1ab1e75a61977a6552e07487d1ec7dc1ae565ea1"
Jul 18 01:24:14.548434 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:24:14.548349    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd_odh-ai-gateway-infra(e71ec4a9-12f7-4b33-ae3f-697fd5703bf3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3"
Jul 18 01:24:14.982556 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.982512    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"]
Jul 18 01:24:14.983002 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.982980    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:14.983115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.983005    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:14.983115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.983018    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:14.983115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.983027    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:14.983115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.983037    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:14.983115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.983045    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:14.983362 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.983161    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faf7e80-4bf2-4a62-adef-d7872837ce71" containerName="revoke-keys"
Jul 18 01:24:14.983362 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.983176    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2115a4d2-ed4c-4c21-b1e6-4b88f843901a" containerName="revoke-keys"
Jul 18 01:24:14.983362 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.983188    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a18fa98-6e76-4146-9100-d731d898b871" containerName="revoke-keys"
Jul 18 01:24:14.987603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.987580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"
Jul 18 01:24:14.992381 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:14.992356    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"]
Jul 18 01:24:15.078333 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:15.078297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7909a9-836f-40ff-8650-98f0b2da7f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-cc4kb\" (UID: \"3b7909a9-836f-40ff-8650-98f0b2da7f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"
Jul 18 01:24:15.179495 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:15.179460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7909a9-836f-40ff-8650-98f0b2da7f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-cc4kb\" (UID: \"3b7909a9-836f-40ff-8650-98f0b2da7f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"
Jul 18 01:24:15.180123 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:15.180103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7909a9-836f-40ff-8650-98f0b2da7f7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-cc4kb\" (UID: \"3b7909a9-836f-40ff-8650-98f0b2da7f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"
Jul 18 01:24:15.301494 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:15.301395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"
Jul 18 01:24:15.439888 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:15.439862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"]
Jul 18 01:24:15.441938 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:24:15.441909    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7909a9_836f_40ff_8650_98f0b2da7f7f.slice/crio-ca0ef65481fc9b5817926440405f4b9791d949bf7356c24b6c47c683eb8da353 WatchSource:0}: Error finding container ca0ef65481fc9b5817926440405f4b9791d949bf7356c24b6c47c683eb8da353: Status 404 returned error can't find the container with id ca0ef65481fc9b5817926440405f4b9791d949bf7356c24b6c47c683eb8da353
Jul 18 01:24:15.555812 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:15.555720    2563 scope.go:117] "RemoveContainer" containerID="711d31a44871c2282e52ee5f1ab1e75a61977a6552e07487d1ec7dc1ae565ea1"
Jul 18 01:24:15.556244 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:24:15.555980    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd_odh-ai-gateway-infra(e71ec4a9-12f7-4b33-ae3f-697fd5703bf3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3"
Jul 18 01:24:15.556612 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:15.556589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" event={"ID":"3b7909a9-836f-40ff-8650-98f0b2da7f7f","Type":"ContainerStarted","Data":"ca0ef65481fc9b5817926440405f4b9791d949bf7356c24b6c47c683eb8da353"}
Jul 18 01:24:16.563144 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:16.563106    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerID="a288364bd166f1d7cc707b951fe67215bcfa6c0598625437447db94b9834256d" exitCode=6
Jul 18 01:24:16.563568 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:16.563197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" event={"ID":"3b7909a9-836f-40ff-8650-98f0b2da7f7f","Type":"ContainerDied","Data":"a288364bd166f1d7cc707b951fe67215bcfa6c0598625437447db94b9834256d"}
Jul 18 01:24:16.563568 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:16.563343    2563 scope.go:117] "RemoveContainer" containerID="a288364bd166f1d7cc707b951fe67215bcfa6c0598625437447db94b9834256d"
Jul 18 01:24:17.570503 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:17.570472    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerID="8f2860276f12b439c837faa793f283f7366482a984abe5010bcfcf27c5fbb74c" exitCode=6
Jul 18 01:24:17.570958 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:17.570569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" event={"ID":"3b7909a9-836f-40ff-8650-98f0b2da7f7f","Type":"ContainerDied","Data":"8f2860276f12b439c837faa793f283f7366482a984abe5010bcfcf27c5fbb74c"}
Jul 18 01:24:17.570958 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:17.570631    2563 scope.go:117] "RemoveContainer" containerID="a288364bd166f1d7cc707b951fe67215bcfa6c0598625437447db94b9834256d"
Jul 18 01:24:17.570958 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:17.570856    2563 scope.go:117] "RemoveContainer" containerID="8f2860276f12b439c837faa793f283f7366482a984abe5010bcfcf27c5fbb74c"
Jul 18 01:24:17.571139 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:24:17.571092    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-cc4kb_odh-ai-gateway-infra(3b7909a9-836f-40ff-8650-98f0b2da7f7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f"
Jul 18 01:24:18.578235 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:18.578199    2563 scope.go:117] "RemoveContainer" containerID="8f2860276f12b439c837faa793f283f7366482a984abe5010bcfcf27c5fbb74c"
Jul 18 01:24:18.578639 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:24:18.578389    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-cc4kb_odh-ai-gateway-infra(3b7909a9-836f-40ff-8650-98f0b2da7f7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f"
Jul 18 01:24:27.170112 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:27.170074    2563 scope.go:117] "RemoveContainer" containerID="711d31a44871c2282e52ee5f1ab1e75a61977a6552e07487d1ec7dc1ae565ea1"
Jul 18 01:24:27.631534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:27.631497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" event={"ID":"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3","Type":"ContainerStarted","Data":"f788607e371569ae42c80a6be6006f7572a349b16837794e90a262d514ccaa4b"}
Jul 18 01:24:27.647821 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:27.647771    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" podStartSLOduration=15.64775625 podStartE2EDuration="15.64775625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:24:27.645717389 +0000 UTC m=+2637.965694771" watchObservedRunningTime="2026-07-18 01:24:27.64775625 +0000 UTC m=+2637.967733632"
Jul 18 01:24:28.638349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:28.638304    2563 generic.go:356] "Generic (PLEG): container finished" podID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerID="f788607e371569ae42c80a6be6006f7572a349b16837794e90a262d514ccaa4b" exitCode=6
Jul 18 01:24:28.638839 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:28.638375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" event={"ID":"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3","Type":"ContainerDied","Data":"f788607e371569ae42c80a6be6006f7572a349b16837794e90a262d514ccaa4b"}
Jul 18 01:24:28.638839 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:28.638437    2563 scope.go:117] "RemoveContainer" containerID="711d31a44871c2282e52ee5f1ab1e75a61977a6552e07487d1ec7dc1ae565ea1"
Jul 18 01:24:28.638839 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:28.638733    2563 scope.go:117] "RemoveContainer" containerID="f788607e371569ae42c80a6be6006f7572a349b16837794e90a262d514ccaa4b"
Jul 18 01:24:28.639130 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:24:28.638978    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd_odh-ai-gateway-infra(e71ec4a9-12f7-4b33-ae3f-697fd5703bf3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3"
Jul 18 01:24:28.656785 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:28.656748    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"]
Jul 18 01:24:29.795017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:29.794987    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"
Jul 18 01:24:29.809585 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:29.809549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ec4a9-12f7-4b33-ae3f-697fd5703bf3-maas-api-service-ca\") pod \"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3\" (UID: \"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3\") "
Jul 18 01:24:29.809916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:29.809892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71ec4a9-12f7-4b33-ae3f-697fd5703bf3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" (UID: "e71ec4a9-12f7-4b33-ae3f-697fd5703bf3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:24:29.910579 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:29.910488    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ec4a9-12f7-4b33-ae3f-697fd5703bf3-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:24:30.653437 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:30.653399    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"
Jul 18 01:24:30.653620 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:30.653400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd" event={"ID":"e71ec4a9-12f7-4b33-ae3f-697fd5703bf3","Type":"ContainerDied","Data":"21484af8361a2ffbcf19013834c3be344115a6f61761eca2f63582b6a91f80f9"}
Jul 18 01:24:30.653620 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:30.653547    2563 scope.go:117] "RemoveContainer" containerID="f788607e371569ae42c80a6be6006f7572a349b16837794e90a262d514ccaa4b"
Jul 18 01:24:30.671388 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:30.671349    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"]
Jul 18 01:24:30.673519 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:30.673494    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-84srd"]
Jul 18 01:24:32.170203 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.170168    2563 scope.go:117] "RemoveContainer" containerID="8f2860276f12b439c837faa793f283f7366482a984abe5010bcfcf27c5fbb74c"
Jul 18 01:24:32.176039 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.176016    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" path="/var/lib/kubelet/pods/e71ec4a9-12f7-4b33-ae3f-697fd5703bf3/volumes"
Jul 18 01:24:32.666697 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.666658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" event={"ID":"3b7909a9-836f-40ff-8650-98f0b2da7f7f","Type":"ContainerStarted","Data":"4ecf5c0899ce14df6f105f0b142af4e8f553f4546551db54b4ce2aae42a4b762"}
Jul 18 01:24:32.691288 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.691235    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" podStartSLOduration=18.69121715 podStartE2EDuration="18.69121715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:24:32.689184416 +0000 UTC m=+2643.009161799" watchObservedRunningTime="2026-07-18 01:24:32.69121715 +0000 UTC m=+2643.011194533"
Jul 18 01:24:32.821704 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.821621    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"]
Jul 18 01:24:32.822312 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.822288    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:24:32.822358 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.822317    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:24:32.822358 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.822343    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:24:32.822358 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.822351    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:24:32.822481 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.822469    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:24:32.822523 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.822484    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:24:32.826209 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.826189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:32.829846 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.829823    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-46dd73e1-serving-cert\""
Jul 18 01:24:32.839778 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.839745    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"]
Jul 18 01:24:32.938751 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.938716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4sd\" (UniqueName: \"kubernetes.io/projected/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-kube-api-access-kv4sd\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:32.938751 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.938761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:32.938993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:32.938793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-openshift-service-ca\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.039973 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.039933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-openshift-service-ca\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.040223 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.040040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4sd\" (UniqueName: \"kubernetes.io/projected/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-kube-api-access-kv4sd\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.040223 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.040101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.040223 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:24:33.040198    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-serving-cert: secret "maas-api-e2e-mt-46dd73e1-serving-cert" not found
Jul 18 01:24:33.040407 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:24:33.040277    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls podName:9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da nodeName:}" failed. No retries permitted until 2026-07-18 01:24:33.540257517 +0000 UTC m=+2643.860234897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls") pod "maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8" (UID: "9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da") : secret "maas-api-e2e-mt-46dd73e1-serving-cert" not found
Jul 18 01:24:33.040771 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.040742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-openshift-service-ca\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.049421 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.049390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4sd\" (UniqueName: \"kubernetes.io/projected/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-kube-api-access-kv4sd\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.545255 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.545217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.547726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.547702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls\") pod \"maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.674097 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.674049    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerID="4ecf5c0899ce14df6f105f0b142af4e8f553f4546551db54b4ce2aae42a4b762" exitCode=6
Jul 18 01:24:33.674275 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.674124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" event={"ID":"3b7909a9-836f-40ff-8650-98f0b2da7f7f","Type":"ContainerDied","Data":"4ecf5c0899ce14df6f105f0b142af4e8f553f4546551db54b4ce2aae42a4b762"}
Jul 18 01:24:33.674275 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.674180    2563 scope.go:117] "RemoveContainer" containerID="8f2860276f12b439c837faa793f283f7366482a984abe5010bcfcf27c5fbb74c"
Jul 18 01:24:33.674441 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.674421    2563 scope.go:117] "RemoveContainer" containerID="4ecf5c0899ce14df6f105f0b142af4e8f553f4546551db54b4ce2aae42a4b762"
Jul 18 01:24:33.674704 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:24:33.674679    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-cc4kb_odh-ai-gateway-infra(3b7909a9-836f-40ff-8650-98f0b2da7f7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f"
Jul 18 01:24:33.701474 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.701437    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"]
Jul 18 01:24:33.737203 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.737158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:33.886354 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:33.886318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"]
Jul 18 01:24:33.888009 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:24:33.887971    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7ebbd0_fa05_4bc5_9bec_7e1b3dae60da.slice/crio-4c5bf8ecc45dd4286b4751751c6c790dcef3e40ef6cd098d331f55426f813063 WatchSource:0}: Error finding container 4c5bf8ecc45dd4286b4751751c6c790dcef3e40ef6cd098d331f55426f813063: Status 404 returned error can't find the container with id 4c5bf8ecc45dd4286b4751751c6c790dcef3e40ef6cd098d331f55426f813063
Jul 18 01:24:34.685101 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:34.685067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8" event={"ID":"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da","Type":"ContainerStarted","Data":"77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7"}
Jul 18 01:24:34.685420 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:34.685157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8" event={"ID":"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da","Type":"ContainerStarted","Data":"4c5bf8ecc45dd4286b4751751c6c790dcef3e40ef6cd098d331f55426f813063"}
Jul 18 01:24:34.685420 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:34.685183    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:34.720542 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:34.720482    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8" podStartSLOduration=2.267684361 podStartE2EDuration="2.720460376s" totalImagesPullingTime="452.776015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:24:34.720077034 +0000 UTC m=+2645.040054409" watchObservedRunningTime="2026-07-18 01:24:34.720460376 +0000 UTC m=+2645.040437760"
Jul 18 01:24:34.822600 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:34.822574    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"
Jul 18 01:24:34.857564 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:34.857532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7909a9-836f-40ff-8650-98f0b2da7f7f-maas-api-service-ca\") pod \"3b7909a9-836f-40ff-8650-98f0b2da7f7f\" (UID: \"3b7909a9-836f-40ff-8650-98f0b2da7f7f\") "
Jul 18 01:24:34.857877 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:34.857853    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7909a9-836f-40ff-8650-98f0b2da7f7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b7909a9-836f-40ff-8650-98f0b2da7f7f" (UID: "3b7909a9-836f-40ff-8650-98f0b2da7f7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:24:34.958668 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:34.958637    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7909a9-836f-40ff-8650-98f0b2da7f7f-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:24:35.695680 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:35.695639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb" event={"ID":"3b7909a9-836f-40ff-8650-98f0b2da7f7f","Type":"ContainerDied","Data":"ca0ef65481fc9b5817926440405f4b9791d949bf7356c24b6c47c683eb8da353"}
Jul 18 01:24:35.695680 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:35.695671    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"
Jul 18 01:24:35.696235 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:35.695717    2563 scope.go:117] "RemoveContainer" containerID="4ecf5c0899ce14df6f105f0b142af4e8f553f4546551db54b4ce2aae42a4b762"
Jul 18 01:24:35.719218 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:35.719182    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"]
Jul 18 01:24:35.723099 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:35.723068    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-cc4kb"]
Jul 18 01:24:36.175787 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:36.175706    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" path="/var/lib/kubelet/pods/3b7909a9-836f-40ff-8650-98f0b2da7f7f/volumes"
Jul 18 01:24:38.139376 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:38.139336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:24:40.701123 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:40.701091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:24:47.632466 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:24:47.632427    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:25:04.635497 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:04.635450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:25:20.943313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:20.943274    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"]
Jul 18 01:25:20.943837 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:20.943506    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8" podUID="9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" containerName="maas-api" containerID="cri-o://77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7" gracePeriod=30
Jul 18 01:25:21.201037 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.201008    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:25:21.263412 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.263372    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-openshift-service-ca\") pod \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") "
Jul 18 01:25:21.263595 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.263500    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv4sd\" (UniqueName: \"kubernetes.io/projected/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-kube-api-access-kv4sd\") pod \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") "
Jul 18 01:25:21.263595 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.263551    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls\") pod \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\" (UID: \"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da\") "
Jul 18 01:25:21.263760 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.263729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" (UID: "9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:25:21.265688 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.265656    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" (UID: "9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:25:21.265688 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.265673    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-kube-api-access-kv4sd" (OuterVolumeSpecName: "kube-api-access-kv4sd") pod "9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" (UID: "9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da"). InnerVolumeSpecName "kube-api-access-kv4sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:25:21.364427 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.364382    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:25:21.364427 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.364421    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:25:21.364654 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.364436    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4sd\" (UniqueName: \"kubernetes.io/projected/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da-kube-api-access-kv4sd\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:25:21.960313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.960280    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" containerID="77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7" exitCode=0
Jul 18 01:25:21.960746 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.960321    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"
Jul 18 01:25:21.960746 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.960362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8" event={"ID":"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da","Type":"ContainerDied","Data":"77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7"}
Jul 18 01:25:21.960746 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.960415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8" event={"ID":"9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da","Type":"ContainerDied","Data":"4c5bf8ecc45dd4286b4751751c6c790dcef3e40ef6cd098d331f55426f813063"}
Jul 18 01:25:21.960746 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.960439    2563 scope.go:117] "RemoveContainer" containerID="77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7"
Jul 18 01:25:21.970512 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.970491    2563 scope.go:117] "RemoveContainer" containerID="77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7"
Jul 18 01:25:21.970788 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:25:21.970768    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7\": container with ID starting with 77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7 not found: ID does not exist" containerID="77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7"
Jul 18 01:25:21.970829 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.970797    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7"} err="failed to get container status \"77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7\": rpc error: code = NotFound desc = could not find container \"77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7\": container with ID starting with 77b5c7593d302905598ca7fd6eb3d965b02095f954e46b6deda02f62ef4f48d7 not found: ID does not exist"
Jul 18 01:25:21.984695 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.984657    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"]
Jul 18 01:25:21.987219 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:21.987194    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-46dd73e1-58b6f6b798-4bxf8"]
Jul 18 01:25:22.175838 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:22.175763    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" path="/var/lib/kubelet/pods/9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da/volumes"
Jul 18 01:25:24.836361 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:24.836315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:25:52.168644 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.168605    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"]
Jul 18 01:25:52.169086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.168982    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:25:52.169086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.168993    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:25:52.169086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169004    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:25:52.169086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169010    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:25:52.169086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169019    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:25:52.169086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169024    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:25:52.169086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169045    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" containerName="maas-api"
Jul 18 01:25:52.169086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169050    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" containerName="maas-api"
Jul 18 01:25:52.169353 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169120    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:25:52.169353 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169130    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:25:52.169353 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169137    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7ebbd0-fa05-4bc5-9bec-7e1b3dae60da" containerName="maas-api"
Jul 18 01:25:52.169353 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169143    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:25:52.169353 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.169149    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71ec4a9-12f7-4b33-ae3f-697fd5703bf3" containerName="revoke-keys"
Jul 18 01:25:52.172210 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.172186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"
Jul 18 01:25:52.174765 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.174700    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:25:52.189744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.189707    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"]
Jul 18 01:25:52.243031 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.242982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e041e0-1b65-428a-ba16-beaa9b64cdf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w\" (UID: \"73e041e0-1b65-428a-ba16-beaa9b64cdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"
Jul 18 01:25:52.344476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.344431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e041e0-1b65-428a-ba16-beaa9b64cdf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w\" (UID: \"73e041e0-1b65-428a-ba16-beaa9b64cdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"
Jul 18 01:25:52.345130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.345106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e041e0-1b65-428a-ba16-beaa9b64cdf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w\" (UID: \"73e041e0-1b65-428a-ba16-beaa9b64cdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"
Jul 18 01:25:52.494858 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.494816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"
Jul 18 01:25:52.633697 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:52.633668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"]
Jul 18 01:25:52.635811 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:25:52.635780    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e041e0_1b65_428a_ba16_beaa9b64cdf0.slice/crio-4e02d71b3223cb1384e97160011793d9e8d4dfb3fb6b1f5ca0310875b16090db WatchSource:0}: Error finding container 4e02d71b3223cb1384e97160011793d9e8d4dfb3fb6b1f5ca0310875b16090db: Status 404 returned error can't find the container with id 4e02d71b3223cb1384e97160011793d9e8d4dfb3fb6b1f5ca0310875b16090db
Jul 18 01:25:53.134597 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:53.134494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" event={"ID":"73e041e0-1b65-428a-ba16-beaa9b64cdf0","Type":"ContainerStarted","Data":"d4dbc18362c35490950352de7127a5bc153d30878bf84dadc407cb31cf6a3a70"}
Jul 18 01:25:53.134597 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:53.134553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" event={"ID":"73e041e0-1b65-428a-ba16-beaa9b64cdf0","Type":"ContainerStarted","Data":"4e02d71b3223cb1384e97160011793d9e8d4dfb3fb6b1f5ca0310875b16090db"}
Jul 18 01:25:53.151594 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:53.151541    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" podStartSLOduration=1.151525787 podStartE2EDuration="1.151525787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:25:53.149407026 +0000 UTC m=+2723.469384409" watchObservedRunningTime="2026-07-18 01:25:53.151525787 +0000 UTC m=+2723.471503170"
Jul 18 01:25:54.142027 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:54.141993    2563 generic.go:356] "Generic (PLEG): container finished" podID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerID="d4dbc18362c35490950352de7127a5bc153d30878bf84dadc407cb31cf6a3a70" exitCode=6
Jul 18 01:25:54.142462 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:54.142032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" event={"ID":"73e041e0-1b65-428a-ba16-beaa9b64cdf0","Type":"ContainerDied","Data":"d4dbc18362c35490950352de7127a5bc153d30878bf84dadc407cb31cf6a3a70"}
Jul 18 01:25:54.142462 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:54.142236    2563 scope.go:117] "RemoveContainer" containerID="d4dbc18362c35490950352de7127a5bc153d30878bf84dadc407cb31cf6a3a70"
Jul 18 01:25:55.149456 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:55.149420    2563 generic.go:356] "Generic (PLEG): container finished" podID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerID="3424a7d676b9204ca54c899251e7a6472d061052e910267b3093add4709c48a4" exitCode=6
Jul 18 01:25:55.149894 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:55.149475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" event={"ID":"73e041e0-1b65-428a-ba16-beaa9b64cdf0","Type":"ContainerDied","Data":"3424a7d676b9204ca54c899251e7a6472d061052e910267b3093add4709c48a4"}
Jul 18 01:25:55.149894 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:55.149534    2563 scope.go:117] "RemoveContainer" containerID="d4dbc18362c35490950352de7127a5bc153d30878bf84dadc407cb31cf6a3a70"
Jul 18 01:25:55.149894 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:55.149727    2563 scope.go:117] "RemoveContainer" containerID="3424a7d676b9204ca54c899251e7a6472d061052e910267b3093add4709c48a4"
Jul 18 01:25:55.150014 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:25:55.149975    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w_odh-ai-gateway-infra(73e041e0-1b65-428a-ba16-beaa9b64cdf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0"
Jul 18 01:25:56.157596 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:25:56.157567    2563 scope.go:117] "RemoveContainer" containerID="3424a7d676b9204ca54c899251e7a6472d061052e910267b3093add4709c48a4"
Jul 18 01:25:56.157974 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:25:56.157761    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w_odh-ai-gateway-infra(73e041e0-1b65-428a-ba16-beaa9b64cdf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0"
Jul 18 01:26:11.169650 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:11.169608    2563 scope.go:117] "RemoveContainer" containerID="3424a7d676b9204ca54c899251e7a6472d061052e910267b3093add4709c48a4"
Jul 18 01:26:12.244954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:12.244922    2563 generic.go:356] "Generic (PLEG): container finished" podID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerID="386cd05f3547f070fdbc3017998fff737d3cb89ea341b052c4de08ec749a6f84" exitCode=6
Jul 18 01:26:12.245606 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:12.244997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" event={"ID":"73e041e0-1b65-428a-ba16-beaa9b64cdf0","Type":"ContainerDied","Data":"386cd05f3547f070fdbc3017998fff737d3cb89ea341b052c4de08ec749a6f84"}
Jul 18 01:26:12.245606 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:12.245084    2563 scope.go:117] "RemoveContainer" containerID="3424a7d676b9204ca54c899251e7a6472d061052e910267b3093add4709c48a4"
Jul 18 01:26:12.245606 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:12.245354    2563 scope.go:117] "RemoveContainer" containerID="386cd05f3547f070fdbc3017998fff737d3cb89ea341b052c4de08ec749a6f84"
Jul 18 01:26:12.245773 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:26:12.245584    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w_odh-ai-gateway-infra(73e041e0-1b65-428a-ba16-beaa9b64cdf0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0"
Jul 18 01:26:13.268699 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:13.268666    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"]
Jul 18 01:26:13.407185 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:13.407155    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"
Jul 18 01:26:13.523845 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:13.523750    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e041e0-1b65-428a-ba16-beaa9b64cdf0-maas-api-service-ca\") pod \"73e041e0-1b65-428a-ba16-beaa9b64cdf0\" (UID: \"73e041e0-1b65-428a-ba16-beaa9b64cdf0\") "
Jul 18 01:26:13.524171 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:13.524146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e041e0-1b65-428a-ba16-beaa9b64cdf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73e041e0-1b65-428a-ba16-beaa9b64cdf0" (UID: "73e041e0-1b65-428a-ba16-beaa9b64cdf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:26:13.625034 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:13.624994    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e041e0-1b65-428a-ba16-beaa9b64cdf0-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:26:14.258183 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:14.258151    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"
Jul 18 01:26:14.258183 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:14.258173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w" event={"ID":"73e041e0-1b65-428a-ba16-beaa9b64cdf0","Type":"ContainerDied","Data":"4e02d71b3223cb1384e97160011793d9e8d4dfb3fb6b1f5ca0310875b16090db"}
Jul 18 01:26:14.258415 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:14.258238    2563 scope.go:117] "RemoveContainer" containerID="386cd05f3547f070fdbc3017998fff737d3cb89ea341b052c4de08ec749a6f84"
Jul 18 01:26:14.274145 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:14.274114    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"]
Jul 18 01:26:14.279916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:14.279891    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xs62w"]
Jul 18 01:26:15.081938 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.081901    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"]
Jul 18 01:26:15.082316 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.082302    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:26:15.082364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.082318    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7909a9-836f-40ff-8650-98f0b2da7f7f" containerName="revoke-keys"
Jul 18 01:26:15.082364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.082335    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:15.082364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.082341    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:15.082364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.082355    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:15.082364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.082364    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:15.082516 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.082426    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:15.082516 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.082433    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:15.086915 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.086894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"
Jul 18 01:26:15.090487 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.090207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:26:15.092527 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.092499    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"]
Jul 18 01:26:15.137016 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.136970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd7198-977c-4535-97bd-f602d38d6560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-v25qm\" (UID: \"5ddd7198-977c-4535-97bd-f602d38d6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"
Jul 18 01:26:15.238265 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.238221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd7198-977c-4535-97bd-f602d38d6560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-v25qm\" (UID: \"5ddd7198-977c-4535-97bd-f602d38d6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"
Jul 18 01:26:15.239009 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.238986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd7198-977c-4535-97bd-f602d38d6560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-v25qm\" (UID: \"5ddd7198-977c-4535-97bd-f602d38d6560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"
Jul 18 01:26:15.399282 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.399191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"
Jul 18 01:26:15.534015 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:15.533987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"]
Jul 18 01:26:15.536071 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:26:15.536029    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddd7198_977c_4535_97bd_f602d38d6560.slice/crio-cf3778235d58dcdb13cb3cbd2d0e5a3b79ef1d2a10098feacc8e2eecd9257248 WatchSource:0}: Error finding container cf3778235d58dcdb13cb3cbd2d0e5a3b79ef1d2a10098feacc8e2eecd9257248: Status 404 returned error can't find the container with id cf3778235d58dcdb13cb3cbd2d0e5a3b79ef1d2a10098feacc8e2eecd9257248
Jul 18 01:26:16.175644 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:16.175610    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" path="/var/lib/kubelet/pods/73e041e0-1b65-428a-ba16-beaa9b64cdf0/volumes"
Jul 18 01:26:16.272382 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:16.272345    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ddd7198-977c-4535-97bd-f602d38d6560" containerID="e5931544983330be5ce2a8f585c3c49b8b24b848abd4809c087ac71ab0ed8ab7" exitCode=6
Jul 18 01:26:16.272598 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:16.272427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm" event={"ID":"5ddd7198-977c-4535-97bd-f602d38d6560","Type":"ContainerDied","Data":"e5931544983330be5ce2a8f585c3c49b8b24b848abd4809c087ac71ab0ed8ab7"}
Jul 18 01:26:16.272598 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:16.272485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm" event={"ID":"5ddd7198-977c-4535-97bd-f602d38d6560","Type":"ContainerStarted","Data":"cf3778235d58dcdb13cb3cbd2d0e5a3b79ef1d2a10098feacc8e2eecd9257248"}
Jul 18 01:26:16.272687 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:16.272619    2563 scope.go:117] "RemoveContainer" containerID="e5931544983330be5ce2a8f585c3c49b8b24b848abd4809c087ac71ab0ed8ab7"
Jul 18 01:26:17.280445 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:17.280408    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ddd7198-977c-4535-97bd-f602d38d6560" containerID="d9ba9c1fee1880e9aa959bcac1a0683e992e7b598fb0348535c6980c7bbcdf82" exitCode=6
Jul 18 01:26:17.280954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:17.280470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm" event={"ID":"5ddd7198-977c-4535-97bd-f602d38d6560","Type":"ContainerDied","Data":"d9ba9c1fee1880e9aa959bcac1a0683e992e7b598fb0348535c6980c7bbcdf82"}
Jul 18 01:26:17.280954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:17.280530    2563 scope.go:117] "RemoveContainer" containerID="e5931544983330be5ce2a8f585c3c49b8b24b848abd4809c087ac71ab0ed8ab7"
Jul 18 01:26:17.280954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:17.280728    2563 scope.go:117] "RemoveContainer" containerID="d9ba9c1fee1880e9aa959bcac1a0683e992e7b598fb0348535c6980c7bbcdf82"
Jul 18 01:26:17.281143 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:26:17.281004    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-v25qm_odh-ai-gateway-infra(5ddd7198-977c-4535-97bd-f602d38d6560)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm" podUID="5ddd7198-977c-4535-97bd-f602d38d6560"
Jul 18 01:26:18.287851 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:18.287823    2563 scope.go:117] "RemoveContainer" containerID="d9ba9c1fee1880e9aa959bcac1a0683e992e7b598fb0348535c6980c7bbcdf82"
Jul 18 01:26:18.288262 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:26:18.288027    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-v25qm_odh-ai-gateway-infra(5ddd7198-977c-4535-97bd-f602d38d6560)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm" podUID="5ddd7198-977c-4535-97bd-f602d38d6560"
Jul 18 01:26:22.149453 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:22.149417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:26:29.062293 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.062250    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"]
Jul 18 01:26:29.064624 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.062641    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:29.064624 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.062653    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:29.064624 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.062721    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e041e0-1b65-428a-ba16-beaa9b64cdf0" containerName="revoke-keys"
Jul 18 01:26:29.065758 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.065739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"
Jul 18 01:26:29.073130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.073077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"]
Jul 18 01:26:29.161236 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.161192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f47fbb-bcd5-4099-81c5-f04a1196772c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8\" (UID: \"26f47fbb-bcd5-4099-81c5-f04a1196772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"
Jul 18 01:26:29.262825 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.262777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f47fbb-bcd5-4099-81c5-f04a1196772c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8\" (UID: \"26f47fbb-bcd5-4099-81c5-f04a1196772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"
Jul 18 01:26:29.263516 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.263494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f47fbb-bcd5-4099-81c5-f04a1196772c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8\" (UID: \"26f47fbb-bcd5-4099-81c5-f04a1196772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"
Jul 18 01:26:29.377832 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.377737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"
Jul 18 01:26:29.519140 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:29.519110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"]
Jul 18 01:26:29.521260 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:26:29.521229    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f47fbb_bcd5_4099_81c5_f04a1196772c.slice/crio-503dabe9e5c20264c1cee6680ed0768108b4d36140484a86d8ada07ad121ee1f WatchSource:0}: Error finding container 503dabe9e5c20264c1cee6680ed0768108b4d36140484a86d8ada07ad121ee1f: Status 404 returned error can't find the container with id 503dabe9e5c20264c1cee6680ed0768108b4d36140484a86d8ada07ad121ee1f
Jul 18 01:26:30.357320 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:30.357222    2563 generic.go:356] "Generic (PLEG): container finished" podID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerID="cef8d52a02abef8557f7b4d8021498f5cd0beb29d7ca93963098ced889305bc5" exitCode=6
Jul 18 01:26:30.357320 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:30.357304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" event={"ID":"26f47fbb-bcd5-4099-81c5-f04a1196772c","Type":"ContainerDied","Data":"cef8d52a02abef8557f7b4d8021498f5cd0beb29d7ca93963098ced889305bc5"}
Jul 18 01:26:30.357702 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:30.357358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" event={"ID":"26f47fbb-bcd5-4099-81c5-f04a1196772c","Type":"ContainerStarted","Data":"503dabe9e5c20264c1cee6680ed0768108b4d36140484a86d8ada07ad121ee1f"}
Jul 18 01:26:30.357702 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:30.357491    2563 scope.go:117] "RemoveContainer" containerID="cef8d52a02abef8557f7b4d8021498f5cd0beb29d7ca93963098ced889305bc5"
Jul 18 01:26:31.364956 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:31.364861    2563 generic.go:356] "Generic (PLEG): container finished" podID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerID="0f62b5636f0506a830815749696d5582e706481c242225794fa780d69bd8e3b6" exitCode=6
Jul 18 01:26:31.365424 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:31.364951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" event={"ID":"26f47fbb-bcd5-4099-81c5-f04a1196772c","Type":"ContainerDied","Data":"0f62b5636f0506a830815749696d5582e706481c242225794fa780d69bd8e3b6"}
Jul 18 01:26:31.365424 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:31.365023    2563 scope.go:117] "RemoveContainer" containerID="cef8d52a02abef8557f7b4d8021498f5cd0beb29d7ca93963098ced889305bc5"
Jul 18 01:26:31.365424 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:31.365183    2563 scope.go:117] "RemoveContainer" containerID="0f62b5636f0506a830815749696d5582e706481c242225794fa780d69bd8e3b6"
Jul 18 01:26:31.365424 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:26:31.365411    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8_odh-ai-gateway-infra(26f47fbb-bcd5-4099-81c5-f04a1196772c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c"
Jul 18 01:26:32.170123 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:32.170087    2563 scope.go:117] "RemoveContainer" containerID="d9ba9c1fee1880e9aa959bcac1a0683e992e7b598fb0348535c6980c7bbcdf82"
Jul 18 01:26:32.372599 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:32.372569    2563 scope.go:117] "RemoveContainer" containerID="0f62b5636f0506a830815749696d5582e706481c242225794fa780d69bd8e3b6"
Jul 18 01:26:32.372983 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:26:32.372770    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8_odh-ai-gateway-infra(26f47fbb-bcd5-4099-81c5-f04a1196772c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c"
Jul 18 01:26:33.379875 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:33.379839    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ddd7198-977c-4535-97bd-f602d38d6560" containerID="d4ab76e4b1f32682cfcd15e5ebc59b6ab0ecbac09eef7557d229155bc0bcd4b8" exitCode=6
Jul 18 01:26:33.380345 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:33.379907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm" event={"ID":"5ddd7198-977c-4535-97bd-f602d38d6560","Type":"ContainerDied","Data":"d4ab76e4b1f32682cfcd15e5ebc59b6ab0ecbac09eef7557d229155bc0bcd4b8"}
Jul 18 01:26:33.380345 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:33.379974    2563 scope.go:117] "RemoveContainer" containerID="d9ba9c1fee1880e9aa959bcac1a0683e992e7b598fb0348535c6980c7bbcdf82"
Jul 18 01:26:33.380345 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:33.380315    2563 scope.go:117] "RemoveContainer" containerID="d4ab76e4b1f32682cfcd15e5ebc59b6ab0ecbac09eef7557d229155bc0bcd4b8"
Jul 18 01:26:33.380583 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:26:33.380548    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-v25qm_odh-ai-gateway-infra(5ddd7198-977c-4535-97bd-f602d38d6560)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm" podUID="5ddd7198-977c-4535-97bd-f602d38d6560"
Jul 18 01:26:34.405832 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:34.405794    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"]
Jul 18 01:26:34.532163 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:34.532134    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"
Jul 18 01:26:34.611829 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:34.611792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd7198-977c-4535-97bd-f602d38d6560-maas-api-service-ca\") pod \"5ddd7198-977c-4535-97bd-f602d38d6560\" (UID: \"5ddd7198-977c-4535-97bd-f602d38d6560\") "
Jul 18 01:26:34.612183 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:34.612157    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddd7198-977c-4535-97bd-f602d38d6560-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ddd7198-977c-4535-97bd-f602d38d6560" (UID: "5ddd7198-977c-4535-97bd-f602d38d6560"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:26:34.713236 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:34.713189    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ddd7198-977c-4535-97bd-f602d38d6560-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:26:35.327839 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:35.327799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:26:35.395962 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:35.395928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm" event={"ID":"5ddd7198-977c-4535-97bd-f602d38d6560","Type":"ContainerDied","Data":"cf3778235d58dcdb13cb3cbd2d0e5a3b79ef1d2a10098feacc8e2eecd9257248"}
Jul 18 01:26:35.396167 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:35.395989    2563 scope.go:117] "RemoveContainer" containerID="d4ab76e4b1f32682cfcd15e5ebc59b6ab0ecbac09eef7557d229155bc0bcd4b8"
Jul 18 01:26:35.396167 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:35.395937    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"
Jul 18 01:26:35.418149 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:35.418119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"]
Jul 18 01:26:35.423747 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:35.423723    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-v25qm"]
Jul 18 01:26:36.176014 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:36.175976    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" path="/var/lib/kubelet/pods/5ddd7198-977c-4535-97bd-f602d38d6560/volumes"
Jul 18 01:26:47.169786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:47.169747    2563 scope.go:117] "RemoveContainer" containerID="0f62b5636f0506a830815749696d5582e706481c242225794fa780d69bd8e3b6"
Jul 18 01:26:47.461917 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:47.461885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" event={"ID":"26f47fbb-bcd5-4099-81c5-f04a1196772c","Type":"ContainerStarted","Data":"217fb1e6d7ead788ec31744b0d1583e8af77d8d911e2d5d84b6ee44d7a66d0dc"}
Jul 18 01:26:47.480712 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:47.480658    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" podStartSLOduration=18.480642472 podStartE2EDuration="18.480642472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:26:47.477665413 +0000 UTC m=+2777.797642795" watchObservedRunningTime="2026-07-18 01:26:47.480642472 +0000 UTC m=+2777.800619855"
Jul 18 01:26:48.470578 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:48.470541    2563 generic.go:356] "Generic (PLEG): container finished" podID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerID="217fb1e6d7ead788ec31744b0d1583e8af77d8d911e2d5d84b6ee44d7a66d0dc" exitCode=6
Jul 18 01:26:48.471017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:48.470620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" event={"ID":"26f47fbb-bcd5-4099-81c5-f04a1196772c","Type":"ContainerDied","Data":"217fb1e6d7ead788ec31744b0d1583e8af77d8d911e2d5d84b6ee44d7a66d0dc"}
Jul 18 01:26:48.471017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:48.470709    2563 scope.go:117] "RemoveContainer" containerID="0f62b5636f0506a830815749696d5582e706481c242225794fa780d69bd8e3b6"
Jul 18 01:26:48.471017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:48.470950    2563 scope.go:117] "RemoveContainer" containerID="217fb1e6d7ead788ec31744b0d1583e8af77d8d911e2d5d84b6ee44d7a66d0dc"
Jul 18 01:26:48.471244 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:26:48.471220    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8_odh-ai-gateway-infra(26f47fbb-bcd5-4099-81c5-f04a1196772c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c"
Jul 18 01:26:48.487747 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:48.487719    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"]
Jul 18 01:26:49.616900 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:49.616872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"
Jul 18 01:26:49.752949 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:49.752843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f47fbb-bcd5-4099-81c5-f04a1196772c-maas-api-service-ca\") pod \"26f47fbb-bcd5-4099-81c5-f04a1196772c\" (UID: \"26f47fbb-bcd5-4099-81c5-f04a1196772c\") "
Jul 18 01:26:49.753267 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:49.753241    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f47fbb-bcd5-4099-81c5-f04a1196772c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26f47fbb-bcd5-4099-81c5-f04a1196772c" (UID: "26f47fbb-bcd5-4099-81c5-f04a1196772c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:26:49.854278 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:49.854231    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f47fbb-bcd5-4099-81c5-f04a1196772c-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:26:50.483672 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:50.483638    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"
Jul 18 01:26:50.483873 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:50.483679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8" event={"ID":"26f47fbb-bcd5-4099-81c5-f04a1196772c","Type":"ContainerDied","Data":"503dabe9e5c20264c1cee6680ed0768108b4d36140484a86d8ada07ad121ee1f"}
Jul 18 01:26:50.483873 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:50.483751    2563 scope.go:117] "RemoveContainer" containerID="217fb1e6d7ead788ec31744b0d1583e8af77d8d911e2d5d84b6ee44d7a66d0dc"
Jul 18 01:26:50.501110 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:50.501082    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"]
Jul 18 01:26:50.506299 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:50.506265    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-kvgv8"]
Jul 18 01:26:52.177127 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:26:52.176997    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" path="/var/lib/kubelet/pods/26f47fbb-bcd5-4099-81c5-f04a1196772c/volumes"
Jul 18 01:27:14.723532 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.723495    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"]
Jul 18 01:27:14.724098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:14.724173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724102    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:14.724173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724114    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:14.724173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724122    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:14.724173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724130    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:14.724386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724139    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:14.724386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724295    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:14.724386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724306    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:14.724386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724330    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:14.724386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724338    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:14.724631 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724457    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:14.724631 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724471    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:14.724631 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724482    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:14.724631 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724488    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:14.724631 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.724494    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:14.727615 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.727596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:14.730197 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.730171    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-390947-serving-cert\""
Jul 18 01:27:14.736687 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.736606    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"]
Jul 18 01:27:14.767684 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.767650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40580944-5fa5-4ce7-8e83-821307d90c96-openshift-service-ca\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:14.767837 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.767699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g5nm\" (UniqueName: \"kubernetes.io/projected/40580944-5fa5-4ce7-8e83-821307d90c96-kube-api-access-4g5nm\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:14.767837 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.767785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:14.868896 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.868852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g5nm\" (UniqueName: \"kubernetes.io/projected/40580944-5fa5-4ce7-8e83-821307d90c96-kube-api-access-4g5nm\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:14.869157 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.869002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:14.869157 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.869046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40580944-5fa5-4ce7-8e83-821307d90c96-openshift-service-ca\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:14.869280 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:27:14.869153    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-serving-cert: secret "maas-api-e2e-inf-a-390947-serving-cert" not found
Jul 18 01:27:14.869280 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:27:14.869226    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls podName:40580944-5fa5-4ce7-8e83-821307d90c96 nodeName:}" failed. No retries permitted until 2026-07-18 01:27:15.369207356 +0000 UTC m=+2805.689184723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls") pod "maas-api-e2e-inf-a-390947-69979b756b-cjxsx" (UID: "40580944-5fa5-4ce7-8e83-821307d90c96") : secret "maas-api-e2e-inf-a-390947-serving-cert" not found
Jul 18 01:27:14.869666 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.869643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40580944-5fa5-4ce7-8e83-821307d90c96-openshift-service-ca\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:14.880291 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:14.880256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g5nm\" (UniqueName: \"kubernetes.io/projected/40580944-5fa5-4ce7-8e83-821307d90c96-kube-api-access-4g5nm\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:15.373398 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:15.373365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:15.375756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:15.375726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls\") pod \"maas-api-e2e-inf-a-390947-69979b756b-cjxsx\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:15.641750 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:15.641660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:15.789424 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:15.789389    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"]
Jul 18 01:27:15.791446 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:27:15.791414    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40580944_5fa5_4ce7_8e83_821307d90c96.slice/crio-d073ab60c3425679a691f7c9ddbf7de92ffa645cea14a125a81c082be6452273 WatchSource:0}: Error finding container d073ab60c3425679a691f7c9ddbf7de92ffa645cea14a125a81c082be6452273: Status 404 returned error can't find the container with id d073ab60c3425679a691f7c9ddbf7de92ffa645cea14a125a81c082be6452273
Jul 18 01:27:16.630142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:16.630029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx" event={"ID":"40580944-5fa5-4ce7-8e83-821307d90c96","Type":"ContainerStarted","Data":"0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e"}
Jul 18 01:27:16.630142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:16.630099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx" event={"ID":"40580944-5fa5-4ce7-8e83-821307d90c96","Type":"ContainerStarted","Data":"d073ab60c3425679a691f7c9ddbf7de92ffa645cea14a125a81c082be6452273"}
Jul 18 01:27:16.630142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:16.630127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:16.651109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:16.651048    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx" podStartSLOduration=2.240428706 podStartE2EDuration="2.651030544s" totalImagesPullingTime="410.601838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:16.648646778 +0000 UTC m=+2806.968624160" watchObservedRunningTime="2026-07-18 01:27:16.651030544 +0000 UTC m=+2806.971007954"
Jul 18 01:27:22.640824 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.640796    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:27:22.774851 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.774809    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"]
Jul 18 01:27:22.775399 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.775376    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:22.775399 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.775397    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddd7198-977c-4535-97bd-f602d38d6560" containerName="revoke-keys"
Jul 18 01:27:22.775620 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.775493    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26f47fbb-bcd5-4099-81c5-f04a1196772c" containerName="revoke-keys"
Jul 18 01:27:22.779400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.779375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.782196 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.782170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0a3406-serving-cert\""
Jul 18 01:27:22.791657 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.791113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"]
Jul 18 01:27:22.842488 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.842454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.842664 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.842501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q985f\" (UniqueName: \"kubernetes.io/projected/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-kube-api-access-q985f\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.842664 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.842531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.935716 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.935629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:27:22.943473 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.943432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.943629 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.943485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q985f\" (UniqueName: \"kubernetes.io/projected/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-kube-api-access-q985f\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.943629 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.943540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.944170 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.944145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.946191 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.946160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:22.954345 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:22.954318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q985f\" (UniqueName: \"kubernetes.io/projected/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-kube-api-access-q985f\") pod \"maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:23.095562 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:23.095525    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:23.241386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:23.241359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"]
Jul 18 01:27:23.243327 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:27:23.243299    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a83dd2_0ed4_44fb_a875_9d5d8774e60a.slice/crio-9a02a322a262f63d82bb1e3c0afb714adb41fc70b3f5c065f05b5fa121c72cd5 WatchSource:0}: Error finding container 9a02a322a262f63d82bb1e3c0afb714adb41fc70b3f5c065f05b5fa121c72cd5: Status 404 returned error can't find the container with id 9a02a322a262f63d82bb1e3c0afb714adb41fc70b3f5c065f05b5fa121c72cd5
Jul 18 01:27:23.669861 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:23.669827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g" event={"ID":"62a83dd2-0ed4-44fb-a875-9d5d8774e60a","Type":"ContainerStarted","Data":"9a02a322a262f63d82bb1e3c0afb714adb41fc70b3f5c065f05b5fa121c72cd5"}
Jul 18 01:27:24.676862 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:24.676819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g" event={"ID":"62a83dd2-0ed4-44fb-a875-9d5d8774e60a","Type":"ContainerStarted","Data":"0293888b88c285ee58e10f9000802e5e52a36c50aa3a8a0d44c7b8f76d1339bd"}
Jul 18 01:27:24.677399 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:24.676921    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:24.695167 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:24.695106    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g" podStartSLOduration=2.3174561479999998 podStartE2EDuration="2.695089382s" totalImagesPullingTime="377.633234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:24.692580841 +0000 UTC m=+2815.012558232" watchObservedRunningTime="2026-07-18 01:27:24.695089382 +0000 UTC m=+2815.015066767"
Jul 18 01:27:29.105463 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.105416    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"]
Jul 18 01:27:29.109384 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.109357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"
Jul 18 01:27:29.111775 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.111752    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:27:29.116899 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.116870    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"]
Jul 18 01:27:29.202481 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.202446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba0a8de-8906-4fe6-91d5-d611e3d7a669-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq\" (UID: \"dba0a8de-8906-4fe6-91d5-d611e3d7a669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"
Jul 18 01:27:29.303943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.303900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba0a8de-8906-4fe6-91d5-d611e3d7a669-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq\" (UID: \"dba0a8de-8906-4fe6-91d5-d611e3d7a669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"
Jul 18 01:27:29.304576 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.304547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba0a8de-8906-4fe6-91d5-d611e3d7a669-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq\" (UID: \"dba0a8de-8906-4fe6-91d5-d611e3d7a669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"
Jul 18 01:27:29.422748 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.422659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"
Jul 18 01:27:29.566104 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.566074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"]
Jul 18 01:27:29.568277 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:27:29.568253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba0a8de_8906_4fe6_91d5_d611e3d7a669.slice/crio-1c33a05c4b07214c25eb7e2df26942e0f51b99734809d6fc9197998f01a552f5 WatchSource:0}: Error finding container 1c33a05c4b07214c25eb7e2df26942e0f51b99734809d6fc9197998f01a552f5: Status 404 returned error can't find the container with id 1c33a05c4b07214c25eb7e2df26942e0f51b99734809d6fc9197998f01a552f5
Jul 18 01:27:29.708479 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:29.708441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq" event={"ID":"dba0a8de-8906-4fe6-91d5-d611e3d7a669","Type":"ContainerStarted","Data":"1c33a05c4b07214c25eb7e2df26942e0f51b99734809d6fc9197998f01a552f5"}
Jul 18 01:27:30.691608 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:30.691576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:27:30.716567 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:30.716522    2563 generic.go:356] "Generic (PLEG): container finished" podID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerID="6612770350f1bf68d4d26919002b1fe35fed0164a93684f9a7d855b48a817846" exitCode=6
Jul 18 01:27:30.716772 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:30.716612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq" event={"ID":"dba0a8de-8906-4fe6-91d5-d611e3d7a669","Type":"ContainerDied","Data":"6612770350f1bf68d4d26919002b1fe35fed0164a93684f9a7d855b48a817846"}
Jul 18 01:27:30.716906 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:30.716851    2563 scope.go:117] "RemoveContainer" containerID="6612770350f1bf68d4d26919002b1fe35fed0164a93684f9a7d855b48a817846"
Jul 18 01:27:31.724103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:31.724047    2563 generic.go:356] "Generic (PLEG): container finished" podID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerID="90eb3b39a3d398e2a0ede7b43f605fdae8492d4c8d66b24ef2b55de212c75df2" exitCode=6
Jul 18 01:27:31.724503 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:31.724133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq" event={"ID":"dba0a8de-8906-4fe6-91d5-d611e3d7a669","Type":"ContainerDied","Data":"90eb3b39a3d398e2a0ede7b43f605fdae8492d4c8d66b24ef2b55de212c75df2"}
Jul 18 01:27:31.724503 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:31.724181    2563 scope.go:117] "RemoveContainer" containerID="6612770350f1bf68d4d26919002b1fe35fed0164a93684f9a7d855b48a817846"
Jul 18 01:27:31.724503 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:31.724319    2563 scope.go:117] "RemoveContainer" containerID="90eb3b39a3d398e2a0ede7b43f605fdae8492d4c8d66b24ef2b55de212c75df2"
Jul 18 01:27:31.724652 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:27:31.724569    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq_odh-ai-gateway-infra(dba0a8de-8906-4fe6-91d5-d611e3d7a669)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669"
Jul 18 01:27:32.731417 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:32.731390    2563 scope.go:117] "RemoveContainer" containerID="90eb3b39a3d398e2a0ede7b43f605fdae8492d4c8d66b24ef2b55de212c75df2"
Jul 18 01:27:32.731807 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:27:32.731592    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq_odh-ai-gateway-infra(dba0a8de-8906-4fe6-91d5-d611e3d7a669)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669"
Jul 18 01:27:34.431044 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:34.431006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:27:48.170138 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:48.170097    2563 scope.go:117] "RemoveContainer" containerID="90eb3b39a3d398e2a0ede7b43f605fdae8492d4c8d66b24ef2b55de212c75df2"
Jul 18 01:27:48.824773 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:48.824738    2563 generic.go:356] "Generic (PLEG): container finished" podID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerID="cf979ada307b98aaea59cb6b0b605090e017249ba54aaceef93bdf69ee0801e5" exitCode=6
Jul 18 01:27:48.824909 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:48.824807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq" event={"ID":"dba0a8de-8906-4fe6-91d5-d611e3d7a669","Type":"ContainerDied","Data":"cf979ada307b98aaea59cb6b0b605090e017249ba54aaceef93bdf69ee0801e5"}
Jul 18 01:27:48.824909 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:48.824900    2563 scope.go:117] "RemoveContainer" containerID="90eb3b39a3d398e2a0ede7b43f605fdae8492d4c8d66b24ef2b55de212c75df2"
Jul 18 01:27:48.825180 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:48.825164    2563 scope.go:117] "RemoveContainer" containerID="cf979ada307b98aaea59cb6b0b605090e017249ba54aaceef93bdf69ee0801e5"
Jul 18 01:27:48.825447 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:27:48.825422    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq_odh-ai-gateway-infra(dba0a8de-8906-4fe6-91d5-d611e3d7a669)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669"
Jul 18 01:27:49.131989 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:49.131902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:27:49.850324 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:49.850289    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"]
Jul 18 01:27:50.011605 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.011580    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"
Jul 18 01:27:50.094326 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.094294    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba0a8de-8906-4fe6-91d5-d611e3d7a669-maas-api-service-ca\") pod \"dba0a8de-8906-4fe6-91d5-d611e3d7a669\" (UID: \"dba0a8de-8906-4fe6-91d5-d611e3d7a669\") "
Jul 18 01:27:50.094715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.094682    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba0a8de-8906-4fe6-91d5-d611e3d7a669-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dba0a8de-8906-4fe6-91d5-d611e3d7a669" (UID: "dba0a8de-8906-4fe6-91d5-d611e3d7a669"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:27:50.194897 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.194868    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba0a8de-8906-4fe6-91d5-d611e3d7a669-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:27:50.838920 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.838888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"
Jul 18 01:27:50.839120 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.838923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq" event={"ID":"dba0a8de-8906-4fe6-91d5-d611e3d7a669","Type":"ContainerDied","Data":"1c33a05c4b07214c25eb7e2df26942e0f51b99734809d6fc9197998f01a552f5"}
Jul 18 01:27:50.839120 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.838990    2563 scope.go:117] "RemoveContainer" containerID="cf979ada307b98aaea59cb6b0b605090e017249ba54aaceef93bdf69ee0801e5"
Jul 18 01:27:50.854832 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.854806    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"]
Jul 18 01:27:50.860528 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:50.860502    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-9vzvq"]
Jul 18 01:27:52.175899 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.175810    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" path="/var/lib/kubelet/pods/dba0a8de-8906-4fe6-91d5-d611e3d7a669/volumes"
Jul 18 01:27:52.249514 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.249482    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"]
Jul 18 01:27:52.249945 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.249927    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:27:52.250031 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.249948    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:27:52.250031 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.249998    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:27:52.250031 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.250008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:27:52.250206 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.250119    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:27:52.250206 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.250134    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:27:52.254540 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.254515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"
Jul 18 01:27:52.256948 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.256928    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:27:52.267083 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.266640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"]
Jul 18 01:27:52.415456 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.415418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ffeff-bf63-41d8-b49e-f738b69e4de1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m\" (UID: \"129ffeff-bf63-41d8-b49e-f738b69e4de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"
Jul 18 01:27:52.516787 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.516748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ffeff-bf63-41d8-b49e-f738b69e4de1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m\" (UID: \"129ffeff-bf63-41d8-b49e-f738b69e4de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"
Jul 18 01:27:52.517419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.517398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ffeff-bf63-41d8-b49e-f738b69e4de1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m\" (UID: \"129ffeff-bf63-41d8-b49e-f738b69e4de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"
Jul 18 01:27:52.575144 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.575105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"
Jul 18 01:27:52.713153 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.713122    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"]
Jul 18 01:27:52.715082 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:27:52.715029    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129ffeff_bf63_41d8_b49e_f738b69e4de1.slice/crio-1aced59edab3a00f73aec8cbea40c4e43ad1efb0768ddc08726c5e981f9b86ec WatchSource:0}: Error finding container 1aced59edab3a00f73aec8cbea40c4e43ad1efb0768ddc08726c5e981f9b86ec: Status 404 returned error can't find the container with id 1aced59edab3a00f73aec8cbea40c4e43ad1efb0768ddc08726c5e981f9b86ec
Jul 18 01:27:52.853531 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:52.853482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" event={"ID":"129ffeff-bf63-41d8-b49e-f738b69e4de1","Type":"ContainerStarted","Data":"1aced59edab3a00f73aec8cbea40c4e43ad1efb0768ddc08726c5e981f9b86ec"}
Jul 18 01:27:53.860946 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:53.860910    2563 generic.go:356] "Generic (PLEG): container finished" podID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerID="7668db44d825c0df8b00e4e8f6960298abb93df54f57f8b2a46e60571fc02ef9" exitCode=6
Jul 18 01:27:53.861435 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:53.860997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" event={"ID":"129ffeff-bf63-41d8-b49e-f738b69e4de1","Type":"ContainerDied","Data":"7668db44d825c0df8b00e4e8f6960298abb93df54f57f8b2a46e60571fc02ef9"}
Jul 18 01:27:53.861435 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:53.861206    2563 scope.go:117] "RemoveContainer" containerID="7668db44d825c0df8b00e4e8f6960298abb93df54f57f8b2a46e60571fc02ef9"
Jul 18 01:27:54.876400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:54.876302    2563 generic.go:356] "Generic (PLEG): container finished" podID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerID="113348f9c9856605c3f188531ba4d68d5b0b1a7340dd3539d15fb4818d8b6141" exitCode=6
Jul 18 01:27:54.876400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:54.876370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" event={"ID":"129ffeff-bf63-41d8-b49e-f738b69e4de1","Type":"ContainerDied","Data":"113348f9c9856605c3f188531ba4d68d5b0b1a7340dd3539d15fb4818d8b6141"}
Jul 18 01:27:54.876881 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:54.876435    2563 scope.go:117] "RemoveContainer" containerID="7668db44d825c0df8b00e4e8f6960298abb93df54f57f8b2a46e60571fc02ef9"
Jul 18 01:27:54.876881 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:54.876689    2563 scope.go:117] "RemoveContainer" containerID="113348f9c9856605c3f188531ba4d68d5b0b1a7340dd3539d15fb4818d8b6141"
Jul 18 01:27:54.876978 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:27:54.876892    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m_odh-ai-gateway-infra(129ffeff-bf63-41d8-b49e-f738b69e4de1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1"
Jul 18 01:27:55.886205 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:27:55.886179    2563 scope.go:117] "RemoveContainer" containerID="113348f9c9856605c3f188531ba4d68d5b0b1a7340dd3539d15fb4818d8b6141"
Jul 18 01:27:55.886601 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:27:55.886398    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m_odh-ai-gateway-infra(129ffeff-bf63-41d8-b49e-f738b69e4de1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1"
Jul 18 01:28:00.441436 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:00.441398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:28:07.192422 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.192382    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"]
Jul 18 01:28:07.192894 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.192876    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:28:07.192939 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.192895    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:28:07.193004 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.192994    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba0a8de-8906-4fe6-91d5-d611e3d7a669" containerName="revoke-keys"
Jul 18 01:28:07.197292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.197275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"
Jul 18 01:28:07.201879 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.201847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"]
Jul 18 01:28:07.247022 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.246988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce249e1-6907-41ad-9bbf-80900df9e69e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc\" (UID: \"1ce249e1-6907-41ad-9bbf-80900df9e69e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"
Jul 18 01:28:07.347601 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.347567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce249e1-6907-41ad-9bbf-80900df9e69e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc\" (UID: \"1ce249e1-6907-41ad-9bbf-80900df9e69e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"
Jul 18 01:28:07.348217 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.348198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce249e1-6907-41ad-9bbf-80900df9e69e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc\" (UID: \"1ce249e1-6907-41ad-9bbf-80900df9e69e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"
Jul 18 01:28:07.510114 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.510049    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"
Jul 18 01:28:07.643237 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.643211    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"]
Jul 18 01:28:07.645027 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:28:07.644997    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce249e1_6907_41ad_9bbf_80900df9e69e.slice/crio-7e02e8d534b7bff8afcb982dfd944dd6f3c8871796cb62ced31f83d1ebcf8aed WatchSource:0}: Error finding container 7e02e8d534b7bff8afcb982dfd944dd6f3c8871796cb62ced31f83d1ebcf8aed: Status 404 returned error can't find the container with id 7e02e8d534b7bff8afcb982dfd944dd6f3c8871796cb62ced31f83d1ebcf8aed
Jul 18 01:28:07.956715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.956676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" event={"ID":"1ce249e1-6907-41ad-9bbf-80900df9e69e","Type":"ContainerStarted","Data":"3fce235d03264a10fb8d8c00060dd6df3e2fbfe6a38696c8ee13532c0bd60a86"}
Jul 18 01:28:07.956898 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.956729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" event={"ID":"1ce249e1-6907-41ad-9bbf-80900df9e69e","Type":"ContainerStarted","Data":"7e02e8d534b7bff8afcb982dfd944dd6f3c8871796cb62ced31f83d1ebcf8aed"}
Jul 18 01:28:07.978965 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:07.974170    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" podStartSLOduration=0.974148654 podStartE2EDuration="974.148654ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:28:07.969977263 +0000 UTC m=+2858.289954651" watchObservedRunningTime="2026-07-18 01:28:07.974148654 +0000 UTC m=+2858.294126037"
Jul 18 01:28:08.963557 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:08.963523    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerID="3fce235d03264a10fb8d8c00060dd6df3e2fbfe6a38696c8ee13532c0bd60a86" exitCode=6
Jul 18 01:28:08.963962 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:08.963621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" event={"ID":"1ce249e1-6907-41ad-9bbf-80900df9e69e","Type":"ContainerDied","Data":"3fce235d03264a10fb8d8c00060dd6df3e2fbfe6a38696c8ee13532c0bd60a86"}
Jul 18 01:28:08.963962 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:08.963822    2563 scope.go:117] "RemoveContainer" containerID="3fce235d03264a10fb8d8c00060dd6df3e2fbfe6a38696c8ee13532c0bd60a86"
Jul 18 01:28:09.170458 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:09.170423    2563 scope.go:117] "RemoveContainer" containerID="113348f9c9856605c3f188531ba4d68d5b0b1a7340dd3539d15fb4818d8b6141"
Jul 18 01:28:09.971482 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:09.971440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" event={"ID":"129ffeff-bf63-41d8-b49e-f738b69e4de1","Type":"ContainerStarted","Data":"764f42287bb312021c9578a3952767655e7febd2f9d12f72bbb4530df28588b7"}
Jul 18 01:28:09.973514 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:09.973483    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerID="30b25d9e30deda22b940d0eb0fe0250d0968855bc7e6cb416195021a721618d9" exitCode=6
Jul 18 01:28:09.973643 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:09.973542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" event={"ID":"1ce249e1-6907-41ad-9bbf-80900df9e69e","Type":"ContainerDied","Data":"30b25d9e30deda22b940d0eb0fe0250d0968855bc7e6cb416195021a721618d9"}
Jul 18 01:28:09.973643 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:09.973588    2563 scope.go:117] "RemoveContainer" containerID="3fce235d03264a10fb8d8c00060dd6df3e2fbfe6a38696c8ee13532c0bd60a86"
Jul 18 01:28:09.973794 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:09.973770    2563 scope.go:117] "RemoveContainer" containerID="30b25d9e30deda22b940d0eb0fe0250d0968855bc7e6cb416195021a721618d9"
Jul 18 01:28:09.973993 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:09.973974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc_odh-ai-gateway-infra(1ce249e1-6907-41ad-9bbf-80900df9e69e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e"
Jul 18 01:28:10.037257 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:10.037204    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" podStartSLOduration=18.037186578 podStartE2EDuration="18.037186578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:28:09.999499048 +0000 UTC m=+2860.319476432" watchObservedRunningTime="2026-07-18 01:28:10.037186578 +0000 UTC m=+2860.357163970"
Jul 18 01:28:10.980516 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:10.980483    2563 generic.go:356] "Generic (PLEG): container finished" podID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerID="764f42287bb312021c9578a3952767655e7febd2f9d12f72bbb4530df28588b7" exitCode=6
Jul 18 01:28:10.980946 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:10.980558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" event={"ID":"129ffeff-bf63-41d8-b49e-f738b69e4de1","Type":"ContainerDied","Data":"764f42287bb312021c9578a3952767655e7febd2f9d12f72bbb4530df28588b7"}
Jul 18 01:28:10.980946 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:10.980637    2563 scope.go:117] "RemoveContainer" containerID="113348f9c9856605c3f188531ba4d68d5b0b1a7340dd3539d15fb4818d8b6141"
Jul 18 01:28:10.981130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:10.981018    2563 scope.go:117] "RemoveContainer" containerID="764f42287bb312021c9578a3952767655e7febd2f9d12f72bbb4530df28588b7"
Jul 18 01:28:10.981313 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:10.981287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m_odh-ai-gateway-infra(129ffeff-bf63-41d8-b49e-f738b69e4de1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1"
Jul 18 01:28:10.982637 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:10.982614    2563 scope.go:117] "RemoveContainer" containerID="30b25d9e30deda22b940d0eb0fe0250d0968855bc7e6cb416195021a721618d9"
Jul 18 01:28:10.982812 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:10.982794    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc_odh-ai-gateway-infra(1ce249e1-6907-41ad-9bbf-80900df9e69e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e"
Jul 18 01:28:11.009277 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:11.009237    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"]
Jul 18 01:28:12.133566 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:12.133535    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"
Jul 18 01:28:12.189114 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:12.189085    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ffeff-bf63-41d8-b49e-f738b69e4de1-maas-api-service-ca\") pod \"129ffeff-bf63-41d8-b49e-f738b69e4de1\" (UID: \"129ffeff-bf63-41d8-b49e-f738b69e4de1\") "
Jul 18 01:28:12.189485 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:12.189450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129ffeff-bf63-41d8-b49e-f738b69e4de1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "129ffeff-bf63-41d8-b49e-f738b69e4de1" (UID: "129ffeff-bf63-41d8-b49e-f738b69e4de1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:28:12.290422 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:12.290325    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ffeff-bf63-41d8-b49e-f738b69e4de1-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:28:12.997844 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:12.997815    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"
Jul 18 01:28:12.998089 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:12.997853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m" event={"ID":"129ffeff-bf63-41d8-b49e-f738b69e4de1","Type":"ContainerDied","Data":"1aced59edab3a00f73aec8cbea40c4e43ad1efb0768ddc08726c5e981f9b86ec"}
Jul 18 01:28:12.998089 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:12.997927    2563 scope.go:117] "RemoveContainer" containerID="764f42287bb312021c9578a3952767655e7febd2f9d12f72bbb4530df28588b7"
Jul 18 01:28:13.019515 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:13.019472    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"]
Jul 18 01:28:13.021749 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:13.021728    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-kxw2m"]
Jul 18 01:28:14.176173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:14.176136    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" path="/var/lib/kubelet/pods/129ffeff-bf63-41d8-b49e-f738b69e4de1/volumes"
Jul 18 01:28:15.199335 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.199300    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"]
Jul 18 01:28:15.199703 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.199692    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:15.199758 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.199704    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:15.199758 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.199717    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:15.199758 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.199723    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:15.199853 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.199796    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:15.199853 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.199804    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:15.204365 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.204341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"
Jul 18 01:28:15.209416 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.209384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"]
Jul 18 01:28:15.316679 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.316637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4d2a6b-9773-44fe-b480-c3c09444fda1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-fml4f\" (UID: \"6c4d2a6b-9773-44fe-b480-c3c09444fda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"
Jul 18 01:28:15.417944 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.417900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4d2a6b-9773-44fe-b480-c3c09444fda1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-fml4f\" (UID: \"6c4d2a6b-9773-44fe-b480-c3c09444fda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"
Jul 18 01:28:15.418592 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.418570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4d2a6b-9773-44fe-b480-c3c09444fda1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-fml4f\" (UID: \"6c4d2a6b-9773-44fe-b480-c3c09444fda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"
Jul 18 01:28:15.516155 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.516065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"
Jul 18 01:28:15.659462 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:15.659271    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"]
Jul 18 01:28:15.662192 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:28:15.662166    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4d2a6b_9773_44fe_b480_c3c09444fda1.slice/crio-b402d186cb59ff9e2b8071239937dd21b3055df271873efbfed462f345751fba WatchSource:0}: Error finding container b402d186cb59ff9e2b8071239937dd21b3055df271873efbfed462f345751fba: Status 404 returned error can't find the container with id b402d186cb59ff9e2b8071239937dd21b3055df271873efbfed462f345751fba
Jul 18 01:28:16.016929 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:16.016889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" event={"ID":"6c4d2a6b-9773-44fe-b480-c3c09444fda1","Type":"ContainerStarted","Data":"6442082f805c360a2013714e23359bc598a3fffeb15f28a45130795db9795ad3"}
Jul 18 01:28:16.017145 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:16.016950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" event={"ID":"6c4d2a6b-9773-44fe-b480-c3c09444fda1","Type":"ContainerStarted","Data":"b402d186cb59ff9e2b8071239937dd21b3055df271873efbfed462f345751fba"}
Jul 18 01:28:16.033594 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:16.033538    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" podStartSLOduration=1.033521217 podStartE2EDuration="1.033521217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:28:16.030986168 +0000 UTC m=+2866.350963563" watchObservedRunningTime="2026-07-18 01:28:16.033521217 +0000 UTC m=+2866.353498599"
Jul 18 01:28:17.023881 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:17.023847    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerID="6442082f805c360a2013714e23359bc598a3fffeb15f28a45130795db9795ad3" exitCode=6
Jul 18 01:28:17.024265 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:17.023930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" event={"ID":"6c4d2a6b-9773-44fe-b480-c3c09444fda1","Type":"ContainerDied","Data":"6442082f805c360a2013714e23359bc598a3fffeb15f28a45130795db9795ad3"}
Jul 18 01:28:17.024265 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:17.024093    2563 scope.go:117] "RemoveContainer" containerID="6442082f805c360a2013714e23359bc598a3fffeb15f28a45130795db9795ad3"
Jul 18 01:28:17.739139 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:17.739104    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:28:18.031881 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:18.031847    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerID="2f304499aae496420c65f2a942efdc0063d426f5fced09c8251add4e4b12b799" exitCode=6
Jul 18 01:28:18.032344 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:18.031937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" event={"ID":"6c4d2a6b-9773-44fe-b480-c3c09444fda1","Type":"ContainerDied","Data":"2f304499aae496420c65f2a942efdc0063d426f5fced09c8251add4e4b12b799"}
Jul 18 01:28:18.032344 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:18.032007    2563 scope.go:117] "RemoveContainer" containerID="6442082f805c360a2013714e23359bc598a3fffeb15f28a45130795db9795ad3"
Jul 18 01:28:18.032344 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:18.032256    2563 scope.go:117] "RemoveContainer" containerID="2f304499aae496420c65f2a942efdc0063d426f5fced09c8251add4e4b12b799"
Jul 18 01:28:18.032513 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:18.032454    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-fml4f_odh-ai-gateway-infra(6c4d2a6b-9773-44fe-b480-c3c09444fda1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1"
Jul 18 01:28:19.040005 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:19.039973    2563 scope.go:117] "RemoveContainer" containerID="2f304499aae496420c65f2a942efdc0063d426f5fced09c8251add4e4b12b799"
Jul 18 01:28:19.040425 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:19.040206    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-fml4f_odh-ai-gateway-infra(6c4d2a6b-9773-44fe-b480-c3c09444fda1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1"
Jul 18 01:28:22.170038 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:22.169957    2563 scope.go:117] "RemoveContainer" containerID="30b25d9e30deda22b940d0eb0fe0250d0968855bc7e6cb416195021a721618d9"
Jul 18 01:28:23.064249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:23.064208    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerID="0be9164e371bdc5818a1e19acac03f61c0f1a4fd4b48d8ef76793170bc662d85" exitCode=6
Jul 18 01:28:23.064450 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:23.064251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" event={"ID":"1ce249e1-6907-41ad-9bbf-80900df9e69e","Type":"ContainerDied","Data":"0be9164e371bdc5818a1e19acac03f61c0f1a4fd4b48d8ef76793170bc662d85"}
Jul 18 01:28:23.064450 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:23.064318    2563 scope.go:117] "RemoveContainer" containerID="30b25d9e30deda22b940d0eb0fe0250d0968855bc7e6cb416195021a721618d9"
Jul 18 01:28:23.064657 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:23.064638    2563 scope.go:117] "RemoveContainer" containerID="0be9164e371bdc5818a1e19acac03f61c0f1a4fd4b48d8ef76793170bc662d85"
Jul 18 01:28:23.064921 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:23.064900    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc_odh-ai-gateway-infra(1ce249e1-6907-41ad-9bbf-80900df9e69e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e"
Jul 18 01:28:24.089638 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:24.089604    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"]
Jul 18 01:28:24.224797 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:24.224765    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"
Jul 18 01:28:24.298467 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:24.298430    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce249e1-6907-41ad-9bbf-80900df9e69e-maas-api-service-ca\") pod \"1ce249e1-6907-41ad-9bbf-80900df9e69e\" (UID: \"1ce249e1-6907-41ad-9bbf-80900df9e69e\") "
Jul 18 01:28:24.298908 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:24.298878    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce249e1-6907-41ad-9bbf-80900df9e69e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ce249e1-6907-41ad-9bbf-80900df9e69e" (UID: "1ce249e1-6907-41ad-9bbf-80900df9e69e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:28:24.399154 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:24.399043    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce249e1-6907-41ad-9bbf-80900df9e69e-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:28:25.078176 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:25.078138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc" event={"ID":"1ce249e1-6907-41ad-9bbf-80900df9e69e","Type":"ContainerDied","Data":"7e02e8d534b7bff8afcb982dfd944dd6f3c8871796cb62ced31f83d1ebcf8aed"}
Jul 18 01:28:25.078176 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:25.078161    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"
Jul 18 01:28:25.078561 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:25.078208    2563 scope.go:117] "RemoveContainer" containerID="0be9164e371bdc5818a1e19acac03f61c0f1a4fd4b48d8ef76793170bc662d85"
Jul 18 01:28:25.102034 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:25.101999    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"]
Jul 18 01:28:25.109841 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:25.109816    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-5d8bc"]
Jul 18 01:28:26.175703 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:26.175667    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" path="/var/lib/kubelet/pods/1ce249e1-6907-41ad-9bbf-80900df9e69e/volumes"
Jul 18 01:28:29.151870 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.151833    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"]
Jul 18 01:28:29.152305 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152245    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:29.152305 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152257    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:29.152305 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152268    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:29.152305 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152274    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:29.152305 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152285    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:29.152305 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152291    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:29.152504 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152357    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:29.152504 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152365    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:29.152504 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152371    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:29.152504 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.152376    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="129ffeff-bf63-41d8-b49e-f738b69e4de1" containerName="revoke-keys"
Jul 18 01:28:29.156691 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.156672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"
Jul 18 01:28:29.162561 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.162524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"]
Jul 18 01:28:29.243856 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.243813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd\" (UID: \"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"
Jul 18 01:28:29.345270 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.345230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd\" (UID: \"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"
Jul 18 01:28:29.345835 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.345815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd\" (UID: \"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"
Jul 18 01:28:29.469955 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.469919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"
Jul 18 01:28:29.611495 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:29.611468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"]
Jul 18 01:28:30.109966 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:30.109868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" event={"ID":"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8","Type":"ContainerStarted","Data":"97cc78a350def95282d4319cb63b066208cb870108acd151779f0e4ced44af7b"}
Jul 18 01:28:30.109966 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:30.109927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" event={"ID":"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8","Type":"ContainerStarted","Data":"a9f07940025b7e43eb6c7ffec2eab934158fe7fcd79cdfa8d20890461383be4c"}
Jul 18 01:28:30.125655 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:30.125599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" podStartSLOduration=1.125582 podStartE2EDuration="1.125582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:28:30.124171007 +0000 UTC m=+2880.444148400" watchObservedRunningTime="2026-07-18 01:28:30.125582 +0000 UTC m=+2880.445559402"
Jul 18 01:28:31.117031 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.116942    2563 generic.go:356] "Generic (PLEG): container finished" podID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerID="97cc78a350def95282d4319cb63b066208cb870108acd151779f0e4ced44af7b" exitCode=6
Jul 18 01:28:31.117412 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.117029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" event={"ID":"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8","Type":"ContainerDied","Data":"97cc78a350def95282d4319cb63b066208cb870108acd151779f0e4ced44af7b"}
Jul 18 01:28:31.117412 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.117240    2563 scope.go:117] "RemoveContainer" containerID="97cc78a350def95282d4319cb63b066208cb870108acd151779f0e4ced44af7b"
Jul 18 01:28:31.757947 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.757905    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"]
Jul 18 01:28:31.758416 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.758401    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:31.758465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.758418    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce249e1-6907-41ad-9bbf-80900df9e69e" containerName="revoke-keys"
Jul 18 01:28:31.761620 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.761604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"
Jul 18 01:28:31.769642 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.769346    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"]
Jul 18 01:28:31.872821 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.872778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e251588-1e23-41a8-8af4-eb1cf199b26e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390947-qwlhg\" (UID: \"1e251588-1e23-41a8-8af4-eb1cf199b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"
Jul 18 01:28:31.973477 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.973428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e251588-1e23-41a8-8af4-eb1cf199b26e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390947-qwlhg\" (UID: \"1e251588-1e23-41a8-8af4-eb1cf199b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"
Jul 18 01:28:31.974169 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:31.974144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e251588-1e23-41a8-8af4-eb1cf199b26e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-390947-qwlhg\" (UID: \"1e251588-1e23-41a8-8af4-eb1cf199b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"
Jul 18 01:28:32.074581 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:32.074528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"
Jul 18 01:28:32.125409 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:32.125369    2563 generic.go:356] "Generic (PLEG): container finished" podID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerID="2cd649065a4f2772ce74cd49697e5acde34c3f71384af01fce04dde5331ad7c2" exitCode=6
Jul 18 01:28:32.125409 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:32.125415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" event={"ID":"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8","Type":"ContainerDied","Data":"2cd649065a4f2772ce74cd49697e5acde34c3f71384af01fce04dde5331ad7c2"}
Jul 18 01:28:32.125943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:32.125471    2563 scope.go:117] "RemoveContainer" containerID="97cc78a350def95282d4319cb63b066208cb870108acd151779f0e4ced44af7b"
Jul 18 01:28:32.125943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:32.125680    2563 scope.go:117] "RemoveContainer" containerID="2cd649065a4f2772ce74cd49697e5acde34c3f71384af01fce04dde5331ad7c2"
Jul 18 01:28:32.125943 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:32.125921    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd_odh-ai-gateway-infra(f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8"
Jul 18 01:28:32.223999 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:32.223912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"]
Jul 18 01:28:32.227706 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:28:32.227672    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e251588_1e23_41a8_8af4_eb1cf199b26e.slice/crio-8d0b0102229f5908f0282aaba31e08e578c7905165047bd1c59512b1cbbe711e WatchSource:0}: Error finding container 8d0b0102229f5908f0282aaba31e08e578c7905165047bd1c59512b1cbbe711e: Status 404 returned error can't find the container with id 8d0b0102229f5908f0282aaba31e08e578c7905165047bd1c59512b1cbbe711e
Jul 18 01:28:33.133027 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:33.132997    2563 scope.go:117] "RemoveContainer" containerID="2cd649065a4f2772ce74cd49697e5acde34c3f71384af01fce04dde5331ad7c2"
Jul 18 01:28:33.133465 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:33.133251    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd_odh-ai-gateway-infra(f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8"
Jul 18 01:28:33.134340 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:33.134315    2563 generic.go:356] "Generic (PLEG): container finished" podID="1e251588-1e23-41a8-8af4-eb1cf199b26e" containerID="d602104da9aa352c791295b306da2cff004d4a88cc532b530def0d72d4d1571e" exitCode=0
Jul 18 01:28:33.134460 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:33.134351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg" event={"ID":"1e251588-1e23-41a8-8af4-eb1cf199b26e","Type":"ContainerDied","Data":"d602104da9aa352c791295b306da2cff004d4a88cc532b530def0d72d4d1571e"}
Jul 18 01:28:33.134460 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:33.134397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg" event={"ID":"1e251588-1e23-41a8-8af4-eb1cf199b26e","Type":"ContainerStarted","Data":"8d0b0102229f5908f0282aaba31e08e578c7905165047bd1c59512b1cbbe711e"}
Jul 18 01:28:34.170017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:34.169978    2563 scope.go:117] "RemoveContainer" containerID="2f304499aae496420c65f2a942efdc0063d426f5fced09c8251add4e4b12b799"
Jul 18 01:28:34.287165 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:34.287138    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"
Jul 18 01:28:34.395295 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:34.395207    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e251588-1e23-41a8-8af4-eb1cf199b26e-maas-api-service-ca\") pod \"1e251588-1e23-41a8-8af4-eb1cf199b26e\" (UID: \"1e251588-1e23-41a8-8af4-eb1cf199b26e\") "
Jul 18 01:28:34.395608 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:34.395581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e251588-1e23-41a8-8af4-eb1cf199b26e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e251588-1e23-41a8-8af4-eb1cf199b26e" (UID: "1e251588-1e23-41a8-8af4-eb1cf199b26e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:28:34.496881 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:34.496853    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e251588-1e23-41a8-8af4-eb1cf199b26e-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:28:35.146335 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:35.146241    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg"
Jul 18 01:28:35.146335 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:35.146268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-390947-qwlhg" event={"ID":"1e251588-1e23-41a8-8af4-eb1cf199b26e","Type":"ContainerDied","Data":"8d0b0102229f5908f0282aaba31e08e578c7905165047bd1c59512b1cbbe711e"}
Jul 18 01:28:35.146335 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:35.146317    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d0b0102229f5908f0282aaba31e08e578c7905165047bd1c59512b1cbbe711e"
Jul 18 01:28:35.148330 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:35.148304    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerID="afa3f9177def01465254d623987136a8dc5c7455b3bbde413fdd1633a54f5f48" exitCode=6
Jul 18 01:28:35.148438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:35.148367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" event={"ID":"6c4d2a6b-9773-44fe-b480-c3c09444fda1","Type":"ContainerDied","Data":"afa3f9177def01465254d623987136a8dc5c7455b3bbde413fdd1633a54f5f48"}
Jul 18 01:28:35.148438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:35.148409    2563 scope.go:117] "RemoveContainer" containerID="2f304499aae496420c65f2a942efdc0063d426f5fced09c8251add4e4b12b799"
Jul 18 01:28:35.148794 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:35.148778    2563 scope.go:117] "RemoveContainer" containerID="afa3f9177def01465254d623987136a8dc5c7455b3bbde413fdd1633a54f5f48"
Jul 18 01:28:35.148996 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:35.148974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-fml4f_odh-ai-gateway-infra(6c4d2a6b-9773-44fe-b480-c3c09444fda1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1"
Jul 18 01:28:36.175312 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:36.175275    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"]
Jul 18 01:28:36.313824 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:36.313800    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"
Jul 18 01:28:36.414154 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:36.414119    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4d2a6b-9773-44fe-b480-c3c09444fda1-maas-api-service-ca\") pod \"6c4d2a6b-9773-44fe-b480-c3c09444fda1\" (UID: \"6c4d2a6b-9773-44fe-b480-c3c09444fda1\") "
Jul 18 01:28:36.414499 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:36.414475    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4d2a6b-9773-44fe-b480-c3c09444fda1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c4d2a6b-9773-44fe-b480-c3c09444fda1" (UID: "6c4d2a6b-9773-44fe-b480-c3c09444fda1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:28:36.514703 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:36.514669    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4d2a6b-9773-44fe-b480-c3c09444fda1-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:28:36.539009 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:36.538970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:28:37.164210 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:37.164182    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"
Jul 18 01:28:37.164210 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:37.164188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f" event={"ID":"6c4d2a6b-9773-44fe-b480-c3c09444fda1","Type":"ContainerDied","Data":"b402d186cb59ff9e2b8071239937dd21b3055df271873efbfed462f345751fba"}
Jul 18 01:28:37.164456 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:37.164272    2563 scope.go:117] "RemoveContainer" containerID="afa3f9177def01465254d623987136a8dc5c7455b3bbde413fdd1633a54f5f48"
Jul 18 01:28:37.190446 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:37.190413    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"]
Jul 18 01:28:37.193081 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:37.193046    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-fml4f"]
Jul 18 01:28:38.184065 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:38.184013    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" path="/var/lib/kubelet/pods/6c4d2a6b-9773-44fe-b480-c3c09444fda1/volumes"
Jul 18 01:28:42.352073 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.352018    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"]
Jul 18 01:28:42.352483 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.352280    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx" podUID="40580944-5fa5-4ce7-8e83-821307d90c96" containerName="maas-api" containerID="cri-o://0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e" gracePeriod=30
Jul 18 01:28:42.613913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.613847    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:28:42.775581 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.775544    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40580944-5fa5-4ce7-8e83-821307d90c96-openshift-service-ca\") pod \"40580944-5fa5-4ce7-8e83-821307d90c96\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") "
Jul 18 01:28:42.775788 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.775609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls\") pod \"40580944-5fa5-4ce7-8e83-821307d90c96\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") "
Jul 18 01:28:42.775788 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.775687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g5nm\" (UniqueName: \"kubernetes.io/projected/40580944-5fa5-4ce7-8e83-821307d90c96-kube-api-access-4g5nm\") pod \"40580944-5fa5-4ce7-8e83-821307d90c96\" (UID: \"40580944-5fa5-4ce7-8e83-821307d90c96\") "
Jul 18 01:28:42.775970 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.775938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40580944-5fa5-4ce7-8e83-821307d90c96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40580944-5fa5-4ce7-8e83-821307d90c96" (UID: "40580944-5fa5-4ce7-8e83-821307d90c96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:28:42.777747 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.777721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40580944-5fa5-4ce7-8e83-821307d90c96" (UID: "40580944-5fa5-4ce7-8e83-821307d90c96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:28:42.777885 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.777864    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40580944-5fa5-4ce7-8e83-821307d90c96-kube-api-access-4g5nm" (OuterVolumeSpecName: "kube-api-access-4g5nm") pod "40580944-5fa5-4ce7-8e83-821307d90c96" (UID: "40580944-5fa5-4ce7-8e83-821307d90c96"). InnerVolumeSpecName "kube-api-access-4g5nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:28:42.877307 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.877213    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g5nm\" (UniqueName: \"kubernetes.io/projected/40580944-5fa5-4ce7-8e83-821307d90c96-kube-api-access-4g5nm\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:28:42.877307 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.877247    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40580944-5fa5-4ce7-8e83-821307d90c96-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:28:42.877307 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:42.877258    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40580944-5fa5-4ce7-8e83-821307d90c96-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:28:43.213177 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.213143    2563 generic.go:356] "Generic (PLEG): container finished" podID="40580944-5fa5-4ce7-8e83-821307d90c96" containerID="0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e" exitCode=0
Jul 18 01:28:43.213328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.213182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx" event={"ID":"40580944-5fa5-4ce7-8e83-821307d90c96","Type":"ContainerDied","Data":"0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e"}
Jul 18 01:28:43.213328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.213186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"
Jul 18 01:28:43.213328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.213226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx" event={"ID":"40580944-5fa5-4ce7-8e83-821307d90c96","Type":"ContainerDied","Data":"d073ab60c3425679a691f7c9ddbf7de92ffa645cea14a125a81c082be6452273"}
Jul 18 01:28:43.213328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.213248    2563 scope.go:117] "RemoveContainer" containerID="0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e"
Jul 18 01:28:43.223159 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.223137    2563 scope.go:117] "RemoveContainer" containerID="0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e"
Jul 18 01:28:43.223440 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:43.223419    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e\": container with ID starting with 0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e not found: ID does not exist" containerID="0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e"
Jul 18 01:28:43.223485 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.223451    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e"} err="failed to get container status \"0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e\": rpc error: code = NotFound desc = could not find container \"0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e\": container with ID starting with 0da74060a4295b8b6395a5b88981839df8b4e2aca8f237c9babe11681fcbf83e not found: ID does not exist"
Jul 18 01:28:43.234567 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.234539    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"]
Jul 18 01:28:43.239671 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:43.239641    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-390947-69979b756b-cjxsx"]
Jul 18 01:28:44.176663 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:44.176626    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40580944-5fa5-4ce7-8e83-821307d90c96" path="/var/lib/kubelet/pods/40580944-5fa5-4ce7-8e83-821307d90c96/volumes"
Jul 18 01:28:45.170379 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:45.170340    2563 scope.go:117] "RemoveContainer" containerID="2cd649065a4f2772ce74cd49697e5acde34c3f71384af01fce04dde5331ad7c2"
Jul 18 01:28:46.231930 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:46.231898    2563 generic.go:356] "Generic (PLEG): container finished" podID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerID="b067030f01b49f518ae73b6727c09d98c53042eeedd15efb74a32ec38c4a06fd" exitCode=6
Jul 18 01:28:46.232342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:46.231970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" event={"ID":"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8","Type":"ContainerDied","Data":"b067030f01b49f518ae73b6727c09d98c53042eeedd15efb74a32ec38c4a06fd"}
Jul 18 01:28:46.232342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:46.232034    2563 scope.go:117] "RemoveContainer" containerID="2cd649065a4f2772ce74cd49697e5acde34c3f71384af01fce04dde5331ad7c2"
Jul 18 01:28:46.232342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:46.232286    2563 scope.go:117] "RemoveContainer" containerID="b067030f01b49f518ae73b6727c09d98c53042eeedd15efb74a32ec38c4a06fd"
Jul 18 01:28:46.232546 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:46.232528    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd_odh-ai-gateway-infra(f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8"
Jul 18 01:28:47.258572 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:47.258534    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"]
Jul 18 01:28:47.397224 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:47.397199    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"
Jul 18 01:28:47.517173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:47.517046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8-maas-api-service-ca\") pod \"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8\" (UID: \"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8\") "
Jul 18 01:28:47.517425 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:47.517401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" (UID: "f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:28:47.618266 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:47.618222    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:28:48.138601 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:48.138563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:28:48.245410 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:48.245377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd" event={"ID":"f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8","Type":"ContainerDied","Data":"a9f07940025b7e43eb6c7ffec2eab934158fe7fcd79cdfa8d20890461383be4c"}
Jul 18 01:28:48.245410 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:48.245405    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"
Jul 18 01:28:48.245689 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:48.245434    2563 scope.go:117] "RemoveContainer" containerID="b067030f01b49f518ae73b6727c09d98c53042eeedd15efb74a32ec38c4a06fd"
Jul 18 01:28:48.261649 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:48.261618    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"]
Jul 18 01:28:48.265384 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:48.265359    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-bc4xd"]
Jul 18 01:28:50.176148 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:50.176115    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" path="/var/lib/kubelet/pods/f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8/volumes"
Jul 18 01:28:52.291612 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.291573    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"]
Jul 18 01:28:52.291997 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.291958    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:52.291997 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.291970    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:52.291997 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.291981    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:52.291997 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.291987    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:52.291997 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.291993    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40580944-5fa5-4ce7-8e83-821307d90c96" containerName="maas-api"
Jul 18 01:28:52.291997 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.291999    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="40580944-5fa5-4ce7-8e83-821307d90c96" containerName="maas-api"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292006    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292011    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292018    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e251588-1e23-41a8-8af4-eb1cf199b26e" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292023    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e251588-1e23-41a8-8af4-eb1cf199b26e" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292030    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292045    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292083    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292184    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="40580944-5fa5-4ce7-8e83-821307d90c96" containerName="maas-api"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292193    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292200    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292206    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e251588-1e23-41a8-8af4-eb1cf199b26e" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292213    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292219    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:52.292239 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.292224    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:52.296476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.296456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"
Jul 18 01:28:52.299485 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.299177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:28:52.302294 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.302267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"]
Jul 18 01:28:52.458609 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.458580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899ef907-438f-4f3f-9a0a-63b113c2c047-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w\" (UID: \"899ef907-438f-4f3f-9a0a-63b113c2c047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"
Jul 18 01:28:52.559734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.559635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899ef907-438f-4f3f-9a0a-63b113c2c047-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w\" (UID: \"899ef907-438f-4f3f-9a0a-63b113c2c047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"
Jul 18 01:28:52.560361 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.560343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899ef907-438f-4f3f-9a0a-63b113c2c047-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w\" (UID: \"899ef907-438f-4f3f-9a0a-63b113c2c047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"
Jul 18 01:28:52.609093 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.609040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"
Jul 18 01:28:52.770483 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:52.770450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"]
Jul 18 01:28:52.772136 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:28:52.772105    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899ef907_438f_4f3f_9a0a_63b113c2c047.slice/crio-cb6b39dc581f0ed0bc6b740c5d719875efd906bdea89b854b164adc920050038 WatchSource:0}: Error finding container cb6b39dc581f0ed0bc6b740c5d719875efd906bdea89b854b164adc920050038: Status 404 returned error can't find the container with id cb6b39dc581f0ed0bc6b740c5d719875efd906bdea89b854b164adc920050038
Jul 18 01:28:53.274573 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:53.274535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" event={"ID":"899ef907-438f-4f3f-9a0a-63b113c2c047","Type":"ContainerStarted","Data":"2920c42b0bf6154e97612479acd3f1aa32f82ea83d944b388c6a27e80c1c3b0c"}
Jul 18 01:28:53.274749 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:53.274598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" event={"ID":"899ef907-438f-4f3f-9a0a-63b113c2c047","Type":"ContainerStarted","Data":"cb6b39dc581f0ed0bc6b740c5d719875efd906bdea89b854b164adc920050038"}
Jul 18 01:28:54.281483 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:54.281442    2563 generic.go:356] "Generic (PLEG): container finished" podID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerID="2920c42b0bf6154e97612479acd3f1aa32f82ea83d944b388c6a27e80c1c3b0c" exitCode=6
Jul 18 01:28:54.281932 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:54.281524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" event={"ID":"899ef907-438f-4f3f-9a0a-63b113c2c047","Type":"ContainerDied","Data":"2920c42b0bf6154e97612479acd3f1aa32f82ea83d944b388c6a27e80c1c3b0c"}
Jul 18 01:28:54.281932 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:54.281754    2563 scope.go:117] "RemoveContainer" containerID="2920c42b0bf6154e97612479acd3f1aa32f82ea83d944b388c6a27e80c1c3b0c"
Jul 18 01:28:55.289427 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:55.289336    2563 generic.go:356] "Generic (PLEG): container finished" podID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerID="14672056052b624ddee84e6d6dbc04001a2fb84ab419aa03e220c99696609a01" exitCode=6
Jul 18 01:28:55.289427 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:55.289398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" event={"ID":"899ef907-438f-4f3f-9a0a-63b113c2c047","Type":"ContainerDied","Data":"14672056052b624ddee84e6d6dbc04001a2fb84ab419aa03e220c99696609a01"}
Jul 18 01:28:55.289893 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:55.289468    2563 scope.go:117] "RemoveContainer" containerID="2920c42b0bf6154e97612479acd3f1aa32f82ea83d944b388c6a27e80c1c3b0c"
Jul 18 01:28:55.289893 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:55.289675    2563 scope.go:117] "RemoveContainer" containerID="14672056052b624ddee84e6d6dbc04001a2fb84ab419aa03e220c99696609a01"
Jul 18 01:28:55.289893 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:55.289878    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w_odh-ai-gateway-infra(899ef907-438f-4f3f-9a0a-63b113c2c047)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047"
Jul 18 01:28:56.297364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:56.297331    2563 scope.go:117] "RemoveContainer" containerID="14672056052b624ddee84e6d6dbc04001a2fb84ab419aa03e220c99696609a01"
Jul 18 01:28:56.297854 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:28:56.297545    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w_odh-ai-gateway-infra(899ef907-438f-4f3f-9a0a-63b113c2c047)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047"
Jul 18 01:28:57.236246 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.236207    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"]
Jul 18 01:28:57.236633 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.236619    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:57.236680 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.236634    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4d2a6b-9773-44fe-b480-c3c09444fda1" containerName="revoke-keys"
Jul 18 01:28:57.236715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.236710    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f31d3fe2-f4a0-4d9c-91a0-3f2c4e6935e8" containerName="revoke-keys"
Jul 18 01:28:57.241015 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.240995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"
Jul 18 01:28:57.246292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.246268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"]
Jul 18 01:28:57.307154 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.307119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf804ac6-29fa-4d55-afe2-0a1d76a02371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp\" (UID: \"cf804ac6-29fa-4d55-afe2-0a1d76a02371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"
Jul 18 01:28:57.407593 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.407553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf804ac6-29fa-4d55-afe2-0a1d76a02371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp\" (UID: \"cf804ac6-29fa-4d55-afe2-0a1d76a02371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"
Jul 18 01:28:57.408236 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.408217    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf804ac6-29fa-4d55-afe2-0a1d76a02371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp\" (UID: \"cf804ac6-29fa-4d55-afe2-0a1d76a02371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"
Jul 18 01:28:57.553568 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.553473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"
Jul 18 01:28:57.689893 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.689865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"]
Jul 18 01:28:57.691643 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:28:57.691600    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf804ac6_29fa_4d55_afe2_0a1d76a02371.slice/crio-1d0c864e0cfac60fd980f59093f2547153e8e282a729fa57f4aa152db07ca600 WatchSource:0}: Error finding container 1d0c864e0cfac60fd980f59093f2547153e8e282a729fa57f4aa152db07ca600: Status 404 returned error can't find the container with id 1d0c864e0cfac60fd980f59093f2547153e8e282a729fa57f4aa152db07ca600
Jul 18 01:28:57.845951 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.845843    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"]
Jul 18 01:28:57.850698 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.850671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"
Jul 18 01:28:57.855794 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.855757    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"]
Jul 18 01:28:57.912178 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:57.912137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4775064f-5f91-47af-a762-288e0408e1ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf\" (UID: \"4775064f-5f91-47af-a762-288e0408e1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"
Jul 18 01:28:58.012577 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:58.012543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4775064f-5f91-47af-a762-288e0408e1ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf\" (UID: \"4775064f-5f91-47af-a762-288e0408e1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"
Jul 18 01:28:58.013181 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:58.013163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4775064f-5f91-47af-a762-288e0408e1ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf\" (UID: \"4775064f-5f91-47af-a762-288e0408e1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"
Jul 18 01:28:58.164999 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:58.164896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"
Jul 18 01:28:58.307174 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:58.307147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"]
Jul 18 01:28:58.309180 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:28:58.309154    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4775064f_5f91_47af_a762_288e0408e1ca.slice/crio-04d148b34a3895609f7f1fb06ff9cfb59f4e9e34460364439c95c2d85d62ecb1 WatchSource:0}: Error finding container 04d148b34a3895609f7f1fb06ff9cfb59f4e9e34460364439c95c2d85d62ecb1: Status 404 returned error can't find the container with id 04d148b34a3895609f7f1fb06ff9cfb59f4e9e34460364439c95c2d85d62ecb1
Jul 18 01:28:58.312193 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:58.312165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" event={"ID":"cf804ac6-29fa-4d55-afe2-0a1d76a02371","Type":"ContainerStarted","Data":"f0516f1a9936f93383223d3fb47554104282eefd1c6d4909e6d048b76d134015"}
Jul 18 01:28:58.312290 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:58.312218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" event={"ID":"cf804ac6-29fa-4d55-afe2-0a1d76a02371","Type":"ContainerStarted","Data":"1d0c864e0cfac60fd980f59093f2547153e8e282a729fa57f4aa152db07ca600"}
Jul 18 01:28:58.329137 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:58.329091    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" podStartSLOduration=1.329044209 podStartE2EDuration="1.329044209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:28:58.326705918 +0000 UTC m=+2908.646683302" watchObservedRunningTime="2026-07-18 01:28:58.329044209 +0000 UTC m=+2908.649021614"
Jul 18 01:28:59.318929 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:59.318890    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerID="f0516f1a9936f93383223d3fb47554104282eefd1c6d4909e6d048b76d134015" exitCode=6
Jul 18 01:28:59.319390 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:59.318983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" event={"ID":"cf804ac6-29fa-4d55-afe2-0a1d76a02371","Type":"ContainerDied","Data":"f0516f1a9936f93383223d3fb47554104282eefd1c6d4909e6d048b76d134015"}
Jul 18 01:28:59.319390 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:59.319202    2563 scope.go:117] "RemoveContainer" containerID="f0516f1a9936f93383223d3fb47554104282eefd1c6d4909e6d048b76d134015"
Jul 18 01:28:59.320616 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:59.320586    2563 generic.go:356] "Generic (PLEG): container finished" podID="4775064f-5f91-47af-a762-288e0408e1ca" containerID="7d5a9afd513dc2ebfeb4fff73faa93501fee7d9708d8fc7168fcfb89abb4b1e6" exitCode=0
Jul 18 01:28:59.320713 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:59.320638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf" event={"ID":"4775064f-5f91-47af-a762-288e0408e1ca","Type":"ContainerDied","Data":"7d5a9afd513dc2ebfeb4fff73faa93501fee7d9708d8fc7168fcfb89abb4b1e6"}
Jul 18 01:28:59.320713 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:28:59.320671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf" event={"ID":"4775064f-5f91-47af-a762-288e0408e1ca","Type":"ContainerStarted","Data":"04d148b34a3895609f7f1fb06ff9cfb59f4e9e34460364439c95c2d85d62ecb1"}
Jul 18 01:29:00.328428 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:00.328394    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerID="2e7dd912d8f42f4a35c08610797584f5715b337e32f2716458935b3fe5b5d7ea" exitCode=6
Jul 18 01:29:00.328913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:00.328484    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" event={"ID":"cf804ac6-29fa-4d55-afe2-0a1d76a02371","Type":"ContainerDied","Data":"2e7dd912d8f42f4a35c08610797584f5715b337e32f2716458935b3fe5b5d7ea"}
Jul 18 01:29:00.328913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:00.328557    2563 scope.go:117] "RemoveContainer" containerID="f0516f1a9936f93383223d3fb47554104282eefd1c6d4909e6d048b76d134015"
Jul 18 01:29:00.328913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:00.328694    2563 scope.go:117] "RemoveContainer" containerID="2e7dd912d8f42f4a35c08610797584f5715b337e32f2716458935b3fe5b5d7ea"
Jul 18 01:29:00.328913 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:00.328901    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp_odh-ai-gateway-infra(cf804ac6-29fa-4d55-afe2-0a1d76a02371)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371"
Jul 18 01:29:00.471995 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:00.471966    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"
Jul 18 01:29:00.532163 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:00.532125    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4775064f-5f91-47af-a762-288e0408e1ca-maas-api-service-ca\") pod \"4775064f-5f91-47af-a762-288e0408e1ca\" (UID: \"4775064f-5f91-47af-a762-288e0408e1ca\") "
Jul 18 01:29:00.532561 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:00.532533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4775064f-5f91-47af-a762-288e0408e1ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4775064f-5f91-47af-a762-288e0408e1ca" (UID: "4775064f-5f91-47af-a762-288e0408e1ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:00.633615 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:00.633527    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4775064f-5f91-47af-a762-288e0408e1ca-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:01.335643 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:01.335613    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf"
Jul 18 01:29:01.336039 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:01.335654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf" event={"ID":"4775064f-5f91-47af-a762-288e0408e1ca","Type":"ContainerDied","Data":"04d148b34a3895609f7f1fb06ff9cfb59f4e9e34460364439c95c2d85d62ecb1"}
Jul 18 01:29:01.336039 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:01.335710    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d148b34a3895609f7f1fb06ff9cfb59f4e9e34460364439c95c2d85d62ecb1"
Jul 18 01:29:01.337667 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:01.337646    2563 scope.go:117] "RemoveContainer" containerID="2e7dd912d8f42f4a35c08610797584f5715b337e32f2716458935b3fe5b5d7ea"
Jul 18 01:29:01.337940 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:01.337920    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp_odh-ai-gateway-infra(cf804ac6-29fa-4d55-afe2-0a1d76a02371)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371"
Jul 18 01:29:08.170326 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.170286    2563 scope.go:117] "RemoveContainer" containerID="14672056052b624ddee84e6d6dbc04001a2fb84ab419aa03e220c99696609a01"
Jul 18 01:29:08.225874 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.225834    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"]
Jul 18 01:29:08.226206 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.226153    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g" podUID="62a83dd2-0ed4-44fb-a875-9d5d8774e60a" containerName="maas-api" containerID="cri-o://0293888b88c285ee58e10f9000802e5e52a36c50aa3a8a0d44c7b8f76d1339bd" gracePeriod=30
Jul 18 01:29:08.380616 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.380565    2563 generic.go:356] "Generic (PLEG): container finished" podID="62a83dd2-0ed4-44fb-a875-9d5d8774e60a" containerID="0293888b88c285ee58e10f9000802e5e52a36c50aa3a8a0d44c7b8f76d1339bd" exitCode=0
Jul 18 01:29:08.380817 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.380636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g" event={"ID":"62a83dd2-0ed4-44fb-a875-9d5d8774e60a","Type":"ContainerDied","Data":"0293888b88c285ee58e10f9000802e5e52a36c50aa3a8a0d44c7b8f76d1339bd"}
Jul 18 01:29:08.470657 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.470632    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:29:08.497201 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.497178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-openshift-service-ca\") pod \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") "
Jul 18 01:29:08.497313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.497238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-maas-api-tls\") pod \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") "
Jul 18 01:29:08.497313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.497272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q985f\" (UniqueName: \"kubernetes.io/projected/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-kube-api-access-q985f\") pod \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\" (UID: \"62a83dd2-0ed4-44fb-a875-9d5d8774e60a\") "
Jul 18 01:29:08.497588 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.497562    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62a83dd2-0ed4-44fb-a875-9d5d8774e60a" (UID: "62a83dd2-0ed4-44fb-a875-9d5d8774e60a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:08.499211 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.499173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62a83dd2-0ed4-44fb-a875-9d5d8774e60a" (UID: "62a83dd2-0ed4-44fb-a875-9d5d8774e60a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:29:08.499489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.499466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-kube-api-access-q985f" (OuterVolumeSpecName: "kube-api-access-q985f") pod "62a83dd2-0ed4-44fb-a875-9d5d8774e60a" (UID: "62a83dd2-0ed4-44fb-a875-9d5d8774e60a"). InnerVolumeSpecName "kube-api-access-q985f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:29:08.598040 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.597999    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:08.598040 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.598031    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:08.598040 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:08.598042    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q985f\" (UniqueName: \"kubernetes.io/projected/62a83dd2-0ed4-44fb-a875-9d5d8774e60a-kube-api-access-q985f\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:09.387952 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.387855    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"
Jul 18 01:29:09.387952 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.387872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g" event={"ID":"62a83dd2-0ed4-44fb-a875-9d5d8774e60a","Type":"ContainerDied","Data":"9a02a322a262f63d82bb1e3c0afb714adb41fc70b3f5c065f05b5fa121c72cd5"}
Jul 18 01:29:09.387952 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.387951    2563 scope.go:117] "RemoveContainer" containerID="0293888b88c285ee58e10f9000802e5e52a36c50aa3a8a0d44c7b8f76d1339bd"
Jul 18 01:29:09.390390 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.390364    2563 generic.go:356] "Generic (PLEG): container finished" podID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerID="64beb86ad243805088e8bf21f9f9942795b642482818e409af8cae4438846596" exitCode=6
Jul 18 01:29:09.390528 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.390435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" event={"ID":"899ef907-438f-4f3f-9a0a-63b113c2c047","Type":"ContainerDied","Data":"64beb86ad243805088e8bf21f9f9942795b642482818e409af8cae4438846596"}
Jul 18 01:29:09.390788 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.390770    2563 scope.go:117] "RemoveContainer" containerID="64beb86ad243805088e8bf21f9f9942795b642482818e409af8cae4438846596"
Jul 18 01:29:09.391013 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:09.390993    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w_odh-ai-gateway-infra(899ef907-438f-4f3f-9a0a-63b113c2c047)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047"
Jul 18 01:29:09.399207 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.399189    2563 scope.go:117] "RemoveContainer" containerID="14672056052b624ddee84e6d6dbc04001a2fb84ab419aa03e220c99696609a01"
Jul 18 01:29:09.420504 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.420470    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"]
Jul 18 01:29:09.427658 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:09.427631    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a3406-6fbc65b44f-wn58g"]
Jul 18 01:29:10.176109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:10.176049    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a83dd2-0ed4-44fb-a875-9d5d8774e60a" path="/var/lib/kubelet/pods/62a83dd2-0ed4-44fb-a875-9d5d8774e60a/volumes"
Jul 18 01:29:10.416570 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:10.416533    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"]
Jul 18 01:29:10.555268 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:10.555242    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"
Jul 18 01:29:10.616538 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:10.616492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899ef907-438f-4f3f-9a0a-63b113c2c047-maas-api-service-ca\") pod \"899ef907-438f-4f3f-9a0a-63b113c2c047\" (UID: \"899ef907-438f-4f3f-9a0a-63b113c2c047\") "
Jul 18 01:29:10.616879 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:10.616853    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899ef907-438f-4f3f-9a0a-63b113c2c047-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "899ef907-438f-4f3f-9a0a-63b113c2c047" (UID: "899ef907-438f-4f3f-9a0a-63b113c2c047"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:10.717860 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:10.717822    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899ef907-438f-4f3f-9a0a-63b113c2c047-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:11.405682 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:11.405652    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"
Jul 18 01:29:11.405866 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:11.405657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w" event={"ID":"899ef907-438f-4f3f-9a0a-63b113c2c047","Type":"ContainerDied","Data":"cb6b39dc581f0ed0bc6b740c5d719875efd906bdea89b854b164adc920050038"}
Jul 18 01:29:11.405866 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:11.405790    2563 scope.go:117] "RemoveContainer" containerID="64beb86ad243805088e8bf21f9f9942795b642482818e409af8cae4438846596"
Jul 18 01:29:11.429146 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:11.429119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"]
Jul 18 01:29:11.432135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:11.432110    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-cgr4w"]
Jul 18 01:29:11.537359 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:11.537322    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:29:12.176539 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:12.176505    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" path="/var/lib/kubelet/pods/899ef907-438f-4f3f-9a0a-63b113c2c047/volumes"
Jul 18 01:29:15.170197 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.170165    2563 scope.go:117] "RemoveContainer" containerID="2e7dd912d8f42f4a35c08610797584f5715b337e32f2716458935b3fe5b5d7ea"
Jul 18 01:29:15.241336 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241298    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"]
Jul 18 01:29:15.241695 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241682    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.241737 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241696    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.241737 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241705    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.241737 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241710    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.241737 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241717    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.241737 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241723    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.241737 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a83dd2-0ed4-44fb-a875-9d5d8774e60a" containerName="maas-api"
Jul 18 01:29:15.241916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241740    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a83dd2-0ed4-44fb-a875-9d5d8774e60a" containerName="maas-api"
Jul 18 01:29:15.241916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241756    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4775064f-5f91-47af-a762-288e0408e1ca" containerName="revoke-keys"
Jul 18 01:29:15.241916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241761    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4775064f-5f91-47af-a762-288e0408e1ca" containerName="revoke-keys"
Jul 18 01:29:15.241916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241824    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a83dd2-0ed4-44fb-a875-9d5d8774e60a" containerName="maas-api"
Jul 18 01:29:15.241916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241832    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.241916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241839    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.241916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241844    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4775064f-5f91-47af-a762-288e0408e1ca" containerName="revoke-keys"
Jul 18 01:29:15.241916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.241850    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="899ef907-438f-4f3f-9a0a-63b113c2c047" containerName="revoke-keys"
Jul 18 01:29:15.245981 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.245964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"
Jul 18 01:29:15.251937 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.251906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"]
Jul 18 01:29:15.357975 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.357938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428d667-fcca-4744-9bcc-066e8384c488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-6jzxm\" (UID: \"e428d667-fcca-4744-9bcc-066e8384c488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"
Jul 18 01:29:15.458764 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.458735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428d667-fcca-4744-9bcc-066e8384c488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-6jzxm\" (UID: \"e428d667-fcca-4744-9bcc-066e8384c488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"
Jul 18 01:29:15.459412 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.459392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428d667-fcca-4744-9bcc-066e8384c488-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-6jzxm\" (UID: \"e428d667-fcca-4744-9bcc-066e8384c488\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"
Jul 18 01:29:15.558503 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.558459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"
Jul 18 01:29:15.696982 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:15.696936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"]
Jul 18 01:29:15.704825 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:29:15.704792    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode428d667_fcca_4744_9bcc_066e8384c488.slice/crio-57bb1137689e81f6c13750c741d754d684b9ed26efbb739c2006abc92a6e2977 WatchSource:0}: Error finding container 57bb1137689e81f6c13750c741d754d684b9ed26efbb739c2006abc92a6e2977: Status 404 returned error can't find the container with id 57bb1137689e81f6c13750c741d754d684b9ed26efbb739c2006abc92a6e2977
Jul 18 01:29:16.442005 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:16.441972    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerID="169452388d2c3932c2af7d1b0a6201821aed3bda5236ece122cbb0c15abfc363" exitCode=6
Jul 18 01:29:16.442489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:16.442045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" event={"ID":"cf804ac6-29fa-4d55-afe2-0a1d76a02371","Type":"ContainerDied","Data":"169452388d2c3932c2af7d1b0a6201821aed3bda5236ece122cbb0c15abfc363"}
Jul 18 01:29:16.442489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:16.442126    2563 scope.go:117] "RemoveContainer" containerID="2e7dd912d8f42f4a35c08610797584f5715b337e32f2716458935b3fe5b5d7ea"
Jul 18 01:29:16.442489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:16.442438    2563 scope.go:117] "RemoveContainer" containerID="169452388d2c3932c2af7d1b0a6201821aed3bda5236ece122cbb0c15abfc363"
Jul 18 01:29:16.442700 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:16.442678    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp_odh-ai-gateway-infra(cf804ac6-29fa-4d55-afe2-0a1d76a02371)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371"
Jul 18 01:29:16.443821 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:16.443791    2563 generic.go:356] "Generic (PLEG): container finished" podID="e428d667-fcca-4744-9bcc-066e8384c488" containerID="7518c38d276a233ae51dbf05de84f4c9ade2d2d55c327fcecb19e00d3ddbfa54" exitCode=6
Jul 18 01:29:16.443913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:16.443858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" event={"ID":"e428d667-fcca-4744-9bcc-066e8384c488","Type":"ContainerDied","Data":"7518c38d276a233ae51dbf05de84f4c9ade2d2d55c327fcecb19e00d3ddbfa54"}
Jul 18 01:29:16.443964 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:16.443911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" event={"ID":"e428d667-fcca-4744-9bcc-066e8384c488","Type":"ContainerStarted","Data":"57bb1137689e81f6c13750c741d754d684b9ed26efbb739c2006abc92a6e2977"}
Jul 18 01:29:16.444008 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:16.443990    2563 scope.go:117] "RemoveContainer" containerID="7518c38d276a233ae51dbf05de84f4c9ade2d2d55c327fcecb19e00d3ddbfa54"
Jul 18 01:29:17.453335 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.453302    2563 generic.go:356] "Generic (PLEG): container finished" podID="e428d667-fcca-4744-9bcc-066e8384c488" containerID="8a15bf2ac6e55e5db5f2c7c713453dc82298cf3ac53909df2c2d44f4331c55dc" exitCode=6
Jul 18 01:29:17.453797 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.453389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" event={"ID":"e428d667-fcca-4744-9bcc-066e8384c488","Type":"ContainerDied","Data":"8a15bf2ac6e55e5db5f2c7c713453dc82298cf3ac53909df2c2d44f4331c55dc"}
Jul 18 01:29:17.453797 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.453449    2563 scope.go:117] "RemoveContainer" containerID="7518c38d276a233ae51dbf05de84f4c9ade2d2d55c327fcecb19e00d3ddbfa54"
Jul 18 01:29:17.453797 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.453577    2563 scope.go:117] "RemoveContainer" containerID="8a15bf2ac6e55e5db5f2c7c713453dc82298cf3ac53909df2c2d44f4331c55dc"
Jul 18 01:29:17.453926 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:17.453820    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-6jzxm_odh-ai-gateway-infra(e428d667-fcca-4744-9bcc-066e8384c488)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" podUID="e428d667-fcca-4744-9bcc-066e8384c488"
Jul 18 01:29:17.468279 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.468252    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"]
Jul 18 01:29:17.612405 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.612376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"
Jul 18 01:29:17.679114 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.679041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf804ac6-29fa-4d55-afe2-0a1d76a02371-maas-api-service-ca\") pod \"cf804ac6-29fa-4d55-afe2-0a1d76a02371\" (UID: \"cf804ac6-29fa-4d55-afe2-0a1d76a02371\") "
Jul 18 01:29:17.679445 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.679421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf804ac6-29fa-4d55-afe2-0a1d76a02371-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf804ac6-29fa-4d55-afe2-0a1d76a02371" (UID: "cf804ac6-29fa-4d55-afe2-0a1d76a02371"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:17.779918 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:17.779825    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf804ac6-29fa-4d55-afe2-0a1d76a02371-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:18.460284 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:18.460255    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"
Jul 18 01:29:18.460724 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:18.460257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp" event={"ID":"cf804ac6-29fa-4d55-afe2-0a1d76a02371","Type":"ContainerDied","Data":"1d0c864e0cfac60fd980f59093f2547153e8e282a729fa57f4aa152db07ca600"}
Jul 18 01:29:18.460724 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:18.460402    2563 scope.go:117] "RemoveContainer" containerID="169452388d2c3932c2af7d1b0a6201821aed3bda5236ece122cbb0c15abfc363"
Jul 18 01:29:18.462428 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:18.462408    2563 scope.go:117] "RemoveContainer" containerID="8a15bf2ac6e55e5db5f2c7c713453dc82298cf3ac53909df2c2d44f4331c55dc"
Jul 18 01:29:18.462679 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:18.462647    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-6jzxm_odh-ai-gateway-infra(e428d667-fcca-4744-9bcc-066e8384c488)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" podUID="e428d667-fcca-4744-9bcc-066e8384c488"
Jul 18 01:29:18.493256 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:18.493227    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"]
Jul 18 01:29:18.495497 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:18.495477    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-2z6fp"]
Jul 18 01:29:20.177119 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:20.177082    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" path="/var/lib/kubelet/pods/cf804ac6-29fa-4d55-afe2-0a1d76a02371/volumes"
Jul 18 01:29:27.448153 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448114    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"]
Jul 18 01:29:27.448728 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448644    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.448728 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448662    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.448728 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448678    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.448728 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448684    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.448728 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448704    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.448728 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448714    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.448943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448786    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.448943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448798    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.448943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.448806    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf804ac6-29fa-4d55-afe2-0a1d76a02371" containerName="revoke-keys"
Jul 18 01:29:27.453003 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.452984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:27.456278 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.456254    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0192d7-serving-cert\""
Jul 18 01:29:27.463913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.463882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"]
Jul 18 01:29:27.565034 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.564993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:27.565250 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.565041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zf8g\" (UniqueName: \"kubernetes.io/projected/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-kube-api-access-6zf8g\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:27.565250 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.565218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:27.665930 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.665890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:27.665930 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.665930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zf8g\" (UniqueName: \"kubernetes.io/projected/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-kube-api-access-6zf8g\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:27.666251 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.665996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:27.666251 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:27.666107    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-serving-cert: secret "maas-api-e2e-api-a-0192d7-serving-cert" not found
Jul 18 01:29:27.666251 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:27.666203    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls podName:77d5f5fe-174b-48f7-a7b2-277580f0fb2a nodeName:}" failed. No retries permitted until 2026-07-18 01:29:28.166180812 +0000 UTC m=+2938.486158176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls") pod "maas-api-e2e-api-a-0192d7-74d7947584-8dx55" (UID: "77d5f5fe-174b-48f7-a7b2-277580f0fb2a") : secret "maas-api-e2e-api-a-0192d7-serving-cert" not found
Jul 18 01:29:27.666644 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.666626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:27.675737 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:27.675702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zf8g\" (UniqueName: \"kubernetes.io/projected/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-kube-api-access-6zf8g\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:28.170574 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:28.170535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:28.173029 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:28.173000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls\") pod \"maas-api-e2e-api-a-0192d7-74d7947584-8dx55\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:28.366820 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:28.366779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:28.515233 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:28.515201    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"]
Jul 18 01:29:28.516626 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:29:28.516590    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d5f5fe_174b_48f7_a7b2_277580f0fb2a.slice/crio-429baeb59f9554baa5d506ac4c46e91df71d95f8f65c52ec58f569571572b295 WatchSource:0}: Error finding container 429baeb59f9554baa5d506ac4c46e91df71d95f8f65c52ec58f569571572b295: Status 404 returned error can't find the container with id 429baeb59f9554baa5d506ac4c46e91df71d95f8f65c52ec58f569571572b295
Jul 18 01:29:28.518099 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:28.518079    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 01:29:28.527428 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:28.527401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55" event={"ID":"77d5f5fe-174b-48f7-a7b2-277580f0fb2a","Type":"ContainerStarted","Data":"429baeb59f9554baa5d506ac4c46e91df71d95f8f65c52ec58f569571572b295"}
Jul 18 01:29:29.534424 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:29.534385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55" event={"ID":"77d5f5fe-174b-48f7-a7b2-277580f0fb2a","Type":"ContainerStarted","Data":"a04eaa5bcf0b011725c09bcb425a93ca191494677bbf0cb9e595d6d4b6272759"}
Jul 18 01:29:29.534926 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:29.534519    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:29.552103 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:29.552027    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55" podStartSLOduration=2.053869797 podStartE2EDuration="2.552010705s" totalImagesPullingTime="498.140908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:29:29.550548194 +0000 UTC m=+2939.870525598" watchObservedRunningTime="2026-07-18 01:29:29.552010705 +0000 UTC m=+2939.871988088"
Jul 18 01:29:32.234399 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:32.234364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:29:34.170328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:34.170294    2563 scope.go:117] "RemoveContainer" containerID="8a15bf2ac6e55e5db5f2c7c713453dc82298cf3ac53909df2c2d44f4331c55dc"
Jul 18 01:29:34.565894 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:34.565854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" event={"ID":"e428d667-fcca-4744-9bcc-066e8384c488","Type":"ContainerStarted","Data":"80cc160f9fd35cdad742d3b54dee2dd9667e16771d76f7520d48b45e6d1e886b"}
Jul 18 01:29:34.580846 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:34.580784    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" podStartSLOduration=19.580764771 podStartE2EDuration="19.580764771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:29:34.580367623 +0000 UTC m=+2944.900345034" watchObservedRunningTime="2026-07-18 01:29:34.580764771 +0000 UTC m=+2944.900742156"
Jul 18 01:29:35.517910 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.517864    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"]
Jul 18 01:29:35.521943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.521899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:35.524443 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.524419    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-232dbd-serving-cert\""
Jul 18 01:29:35.531740 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.531714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"]
Jul 18 01:29:35.546405 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.546375    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:35.575466 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.575434    2563 generic.go:356] "Generic (PLEG): container finished" podID="e428d667-fcca-4744-9bcc-066e8384c488" containerID="80cc160f9fd35cdad742d3b54dee2dd9667e16771d76f7520d48b45e6d1e886b" exitCode=6
Jul 18 01:29:35.575635 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.575469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" event={"ID":"e428d667-fcca-4744-9bcc-066e8384c488","Type":"ContainerDied","Data":"80cc160f9fd35cdad742d3b54dee2dd9667e16771d76f7520d48b45e6d1e886b"}
Jul 18 01:29:35.575635 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.575553    2563 scope.go:117] "RemoveContainer" containerID="8a15bf2ac6e55e5db5f2c7c713453dc82298cf3ac53909df2c2d44f4331c55dc"
Jul 18 01:29:35.575873 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.575857    2563 scope.go:117] "RemoveContainer" containerID="80cc160f9fd35cdad742d3b54dee2dd9667e16771d76f7520d48b45e6d1e886b"
Jul 18 01:29:35.576259 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:35.576234    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-6jzxm_odh-ai-gateway-infra(e428d667-fcca-4744-9bcc-066e8384c488)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" podUID="e428d667-fcca-4744-9bcc-066e8384c488"
Jul 18 01:29:35.594559 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.594526    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"]
Jul 18 01:29:35.639258 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.639222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:35.639415 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.639291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:35.639415 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.639309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndk5\" (UniqueName: \"kubernetes.io/projected/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-kube-api-access-qndk5\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:35.740651 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.740615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:35.740855 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.740673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:35.740855 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.740697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndk5\" (UniqueName: \"kubernetes.io/projected/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-kube-api-access-qndk5\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:35.740855 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:35.740802    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-serving-cert: secret "maas-api-e2e-api-b-232dbd-serving-cert" not found
Jul 18 01:29:35.741037 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:35.740878    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls podName:c8b235a3-32b1-40cc-be09-bc20df7f2b6d nodeName:}" failed. No retries permitted until 2026-07-18 01:29:36.240855843 +0000 UTC m=+2946.560833214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls") pod "maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw" (UID: "c8b235a3-32b1-40cc-be09-bc20df7f2b6d") : secret "maas-api-e2e-api-b-232dbd-serving-cert" not found
Jul 18 01:29:35.741427 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.741398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:35.748796 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:35.748768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndk5\" (UniqueName: \"kubernetes.io/projected/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-kube-api-access-qndk5\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:36.246222 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:36.246174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:36.248613 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:36.248588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls\") pod \"maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:36.434750 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:36.434713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:36.587651 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:36.587616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"]
Jul 18 01:29:36.592038 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:29:36.592000    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b235a3_32b1_40cc_be09_bc20df7f2b6d.slice/crio-467e0b5e502d1f31f66c72dc11ad909b21447e9704596cc9f725592df599b84e WatchSource:0}: Error finding container 467e0b5e502d1f31f66c72dc11ad909b21447e9704596cc9f725592df599b84e: Status 404 returned error can't find the container with id 467e0b5e502d1f31f66c72dc11ad909b21447e9704596cc9f725592df599b84e
Jul 18 01:29:36.719990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:36.719963    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"
Jul 18 01:29:36.852050 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:36.851965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428d667-fcca-4744-9bcc-066e8384c488-maas-api-service-ca\") pod \"e428d667-fcca-4744-9bcc-066e8384c488\" (UID: \"e428d667-fcca-4744-9bcc-066e8384c488\") "
Jul 18 01:29:36.852339 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:36.852319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e428d667-fcca-4744-9bcc-066e8384c488-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e428d667-fcca-4744-9bcc-066e8384c488" (UID: "e428d667-fcca-4744-9bcc-066e8384c488"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:36.953245 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:36.953221    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e428d667-fcca-4744-9bcc-066e8384c488-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:37.600652 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.600601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw" event={"ID":"c8b235a3-32b1-40cc-be09-bc20df7f2b6d","Type":"ContainerStarted","Data":"393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666"}
Jul 18 01:29:37.601133 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.600663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw" event={"ID":"c8b235a3-32b1-40cc-be09-bc20df7f2b6d","Type":"ContainerStarted","Data":"467e0b5e502d1f31f66c72dc11ad909b21447e9704596cc9f725592df599b84e"}
Jul 18 01:29:37.601133 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.600720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:37.602002 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.601976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm" event={"ID":"e428d667-fcca-4744-9bcc-066e8384c488","Type":"ContainerDied","Data":"57bb1137689e81f6c13750c741d754d684b9ed26efbb739c2006abc92a6e2977"}
Jul 18 01:29:37.602002 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.601999    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"
Jul 18 01:29:37.602223 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.602036    2563 scope.go:117] "RemoveContainer" containerID="80cc160f9fd35cdad742d3b54dee2dd9667e16771d76f7520d48b45e6d1e886b"
Jul 18 01:29:37.622249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.622204    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw" podStartSLOduration=2.335148067 podStartE2EDuration="2.622189487s" totalImagesPullingTime="287.04142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:29:37.61963066 +0000 UTC m=+2947.939608043" watchObservedRunningTime="2026-07-18 01:29:37.622189487 +0000 UTC m=+2947.942166869"
Jul 18 01:29:37.638528 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.638499    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"]
Jul 18 01:29:37.640742 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:37.640708    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6jzxm"]
Jul 18 01:29:38.175532 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:38.175490    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e428d667-fcca-4744-9bcc-066e8384c488" path="/var/lib/kubelet/pods/e428d667-fcca-4744-9bcc-066e8384c488/volumes"
Jul 18 01:29:42.336247 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336212    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"]
Jul 18 01:29:42.336638 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336621    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:42.336638 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336635    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:42.336715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336646    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:42.336715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336652    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:42.336715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336660    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:42.336715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:42.336831 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336730    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:42.336831 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.336737    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:42.339781 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.339761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"
Jul 18 01:29:42.342659 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.342638    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:29:42.347742 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.347589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"]
Jul 18 01:29:42.503940 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.503904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c6be1-36d1-451b-b9a5-125c92013471-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv\" (UID: \"0b3c6be1-36d1-451b-b9a5-125c92013471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"
Jul 18 01:29:42.605200 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.605110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c6be1-36d1-451b-b9a5-125c92013471-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv\" (UID: \"0b3c6be1-36d1-451b-b9a5-125c92013471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"
Jul 18 01:29:42.605791 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.605766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c6be1-36d1-451b-b9a5-125c92013471-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv\" (UID: \"0b3c6be1-36d1-451b-b9a5-125c92013471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"
Jul 18 01:29:42.677390 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.677344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"
Jul 18 01:29:42.822099 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:42.821992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"]
Jul 18 01:29:42.822822 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:29:42.822786    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3c6be1_36d1_451b_b9a5_125c92013471.slice/crio-8944269b5405361afd1fdeedca49a8d8a672909ab3a784d42312bf79dc0cf483 WatchSource:0}: Error finding container 8944269b5405361afd1fdeedca49a8d8a672909ab3a784d42312bf79dc0cf483: Status 404 returned error can't find the container with id 8944269b5405361afd1fdeedca49a8d8a672909ab3a784d42312bf79dc0cf483
Jul 18 01:29:43.614596 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:43.614566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:29:43.638575 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:43.638530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:29:43.639710 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:43.639680    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerID="60f511529cd639d30cb257bb9937b6087a41aa83e0845466ba1842c72aa46389" exitCode=6
Jul 18 01:29:43.639851 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:43.639760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" event={"ID":"0b3c6be1-36d1-451b-b9a5-125c92013471","Type":"ContainerDied","Data":"60f511529cd639d30cb257bb9937b6087a41aa83e0845466ba1842c72aa46389"}
Jul 18 01:29:43.639851 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:43.639808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" event={"ID":"0b3c6be1-36d1-451b-b9a5-125c92013471","Type":"ContainerStarted","Data":"8944269b5405361afd1fdeedca49a8d8a672909ab3a784d42312bf79dc0cf483"}
Jul 18 01:29:43.639937 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:43.639900    2563 scope.go:117] "RemoveContainer" containerID="60f511529cd639d30cb257bb9937b6087a41aa83e0845466ba1842c72aa46389"
Jul 18 01:29:44.647043 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:44.647004    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerID="3fba0ac7810561fe621fb92e0f0ad60811cc6237e36b03280344f9d8f99e7835" exitCode=6
Jul 18 01:29:44.647539 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:44.647094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" event={"ID":"0b3c6be1-36d1-451b-b9a5-125c92013471","Type":"ContainerDied","Data":"3fba0ac7810561fe621fb92e0f0ad60811cc6237e36b03280344f9d8f99e7835"}
Jul 18 01:29:44.647539 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:44.647161    2563 scope.go:117] "RemoveContainer" containerID="60f511529cd639d30cb257bb9937b6087a41aa83e0845466ba1842c72aa46389"
Jul 18 01:29:44.647539 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:44.647309    2563 scope.go:117] "RemoveContainer" containerID="3fba0ac7810561fe621fb92e0f0ad60811cc6237e36b03280344f9d8f99e7835"
Jul 18 01:29:44.647711 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:44.647563    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv_odh-ai-gateway-infra(0b3c6be1-36d1-451b-b9a5-125c92013471)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471"
Jul 18 01:29:45.654412 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:45.654387    2563 scope.go:117] "RemoveContainer" containerID="3fba0ac7810561fe621fb92e0f0ad60811cc6237e36b03280344f9d8f99e7835"
Jul 18 01:29:45.654786 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:45.654559    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv_odh-ai-gateway-infra(0b3c6be1-36d1-451b-b9a5-125c92013471)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471"
Jul 18 01:29:47.823363 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:47.823328    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"]
Jul 18 01:29:47.823824 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:47.823810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e428d667-fcca-4744-9bcc-066e8384c488" containerName="revoke-keys"
Jul 18 01:29:47.828244 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:47.828224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"
Jul 18 01:29:47.835788 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:47.835756    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"]
Jul 18 01:29:47.955300 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:47.955262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4019fb29-565b-4347-9e83-ae2e1f398447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0192d7-csqtd\" (UID: \"4019fb29-565b-4347-9e83-ae2e1f398447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"
Jul 18 01:29:48.056811 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:48.056767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4019fb29-565b-4347-9e83-ae2e1f398447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0192d7-csqtd\" (UID: \"4019fb29-565b-4347-9e83-ae2e1f398447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"
Jul 18 01:29:48.057465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:48.057444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4019fb29-565b-4347-9e83-ae2e1f398447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0192d7-csqtd\" (UID: \"4019fb29-565b-4347-9e83-ae2e1f398447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"
Jul 18 01:29:48.141934 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:48.141841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"
Jul 18 01:29:48.288536 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:48.288498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"]
Jul 18 01:29:48.291882 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:29:48.291852    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4019fb29_565b_4347_9e83_ae2e1f398447.slice/crio-cf7b124fd0d8f220e3746694b57b50cd7984d57775ead2db61c9b6b38b01b5ce WatchSource:0}: Error finding container cf7b124fd0d8f220e3746694b57b50cd7984d57775ead2db61c9b6b38b01b5ce: Status 404 returned error can't find the container with id cf7b124fd0d8f220e3746694b57b50cd7984d57775ead2db61c9b6b38b01b5ce
Jul 18 01:29:48.672608 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:48.672565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd" event={"ID":"4019fb29-565b-4347-9e83-ae2e1f398447","Type":"ContainerStarted","Data":"8cbd57e3c3cca4536a853414925676eda41b6fe754a77823da27a5cc2daf15a4"}
Jul 18 01:29:48.672830 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:48.672646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd" event={"ID":"4019fb29-565b-4347-9e83-ae2e1f398447","Type":"ContainerStarted","Data":"cf7b124fd0d8f220e3746694b57b50cd7984d57775ead2db61c9b6b38b01b5ce"}
Jul 18 01:29:48.687438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:48.687379    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd" podStartSLOduration=1.6873608930000001 podStartE2EDuration="1.687360893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:29:48.685899078 +0000 UTC m=+2959.005876461" watchObservedRunningTime="2026-07-18 01:29:48.687360893 +0000 UTC m=+2959.007338276"
Jul 18 01:29:49.680509 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:49.680469    2563 generic.go:356] "Generic (PLEG): container finished" podID="4019fb29-565b-4347-9e83-ae2e1f398447" containerID="8cbd57e3c3cca4536a853414925676eda41b6fe754a77823da27a5cc2daf15a4" exitCode=0
Jul 18 01:29:49.680936 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:49.680564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd" event={"ID":"4019fb29-565b-4347-9e83-ae2e1f398447","Type":"ContainerDied","Data":"8cbd57e3c3cca4536a853414925676eda41b6fe754a77823da27a5cc2daf15a4"}
Jul 18 01:29:50.825943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:50.825917    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"
Jul 18 01:29:50.983736 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:50.983696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4019fb29-565b-4347-9e83-ae2e1f398447-maas-api-service-ca\") pod \"4019fb29-565b-4347-9e83-ae2e1f398447\" (UID: \"4019fb29-565b-4347-9e83-ae2e1f398447\") "
Jul 18 01:29:50.984120 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:50.984096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4019fb29-565b-4347-9e83-ae2e1f398447-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4019fb29-565b-4347-9e83-ae2e1f398447" (UID: "4019fb29-565b-4347-9e83-ae2e1f398447"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:51.085347 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:51.085303    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4019fb29-565b-4347-9e83-ae2e1f398447-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:51.694178 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:51.694149    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd"
Jul 18 01:29:51.694377 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:51.694182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0192d7-csqtd" event={"ID":"4019fb29-565b-4347-9e83-ae2e1f398447","Type":"ContainerDied","Data":"cf7b124fd0d8f220e3746694b57b50cd7984d57775ead2db61c9b6b38b01b5ce"}
Jul 18 01:29:51.694377 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:51.694238    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf7b124fd0d8f220e3746694b57b50cd7984d57775ead2db61c9b6b38b01b5ce"
Jul 18 01:29:57.277113 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.277078    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"]
Jul 18 01:29:57.277499 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.277482    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4019fb29-565b-4347-9e83-ae2e1f398447" containerName="revoke-keys"
Jul 18 01:29:57.277499 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.277493    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4019fb29-565b-4347-9e83-ae2e1f398447" containerName="revoke-keys"
Jul 18 01:29:57.277577 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.277572    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4019fb29-565b-4347-9e83-ae2e1f398447" containerName="revoke-keys"
Jul 18 01:29:57.280698 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.280675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"
Jul 18 01:29:57.288554 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.288521    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"]
Jul 18 01:29:57.346263 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.346224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcabcdcc-1620-412a-a8e7-df6e5c157cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c\" (UID: \"bcabcdcc-1620-412a-a8e7-df6e5c157cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"
Jul 18 01:29:57.447236 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.447199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcabcdcc-1620-412a-a8e7-df6e5c157cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c\" (UID: \"bcabcdcc-1620-412a-a8e7-df6e5c157cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"
Jul 18 01:29:57.447881 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.447855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcabcdcc-1620-412a-a8e7-df6e5c157cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c\" (UID: \"bcabcdcc-1620-412a-a8e7-df6e5c157cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"
Jul 18 01:29:57.593047 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.592949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"
Jul 18 01:29:57.729816 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:57.729787    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"]
Jul 18 01:29:57.733261 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:29:57.733219    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcabcdcc_1620_412a_a8e7_df6e5c157cd6.slice/crio-0662ce7dd1daa85a2e95d3f054f4f5d8fc5ae37a1c77af0bec4133798636e3c9 WatchSource:0}: Error finding container 0662ce7dd1daa85a2e95d3f054f4f5d8fc5ae37a1c77af0bec4133798636e3c9: Status 404 returned error can't find the container with id 0662ce7dd1daa85a2e95d3f054f4f5d8fc5ae37a1c77af0bec4133798636e3c9
Jul 18 01:29:58.574204 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.574123    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"]
Jul 18 01:29:58.574749 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.574358    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55" podUID="77d5f5fe-174b-48f7-a7b2-277580f0fb2a" containerName="maas-api" containerID="cri-o://a04eaa5bcf0b011725c09bcb425a93ca191494677bbf0cb9e595d6d4b6272759" gracePeriod=30
Jul 18 01:29:58.745135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.745097    2563 generic.go:356] "Generic (PLEG): container finished" podID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerID="0104f2584baa91ceb0e9ffca4425565aa456fc9dbed28e8997171ff3e250f8b2" exitCode=6
Jul 18 01:29:58.745334 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.745172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" event={"ID":"bcabcdcc-1620-412a-a8e7-df6e5c157cd6","Type":"ContainerDied","Data":"0104f2584baa91ceb0e9ffca4425565aa456fc9dbed28e8997171ff3e250f8b2"}
Jul 18 01:29:58.745334 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.745266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" event={"ID":"bcabcdcc-1620-412a-a8e7-df6e5c157cd6","Type":"ContainerStarted","Data":"0662ce7dd1daa85a2e95d3f054f4f5d8fc5ae37a1c77af0bec4133798636e3c9"}
Jul 18 01:29:58.745456 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.745371    2563 scope.go:117] "RemoveContainer" containerID="0104f2584baa91ceb0e9ffca4425565aa456fc9dbed28e8997171ff3e250f8b2"
Jul 18 01:29:58.748122 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.748092    2563 generic.go:356] "Generic (PLEG): container finished" podID="77d5f5fe-174b-48f7-a7b2-277580f0fb2a" containerID="a04eaa5bcf0b011725c09bcb425a93ca191494677bbf0cb9e595d6d4b6272759" exitCode=0
Jul 18 01:29:58.748249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.748151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55" event={"ID":"77d5f5fe-174b-48f7-a7b2-277580f0fb2a","Type":"ContainerDied","Data":"a04eaa5bcf0b011725c09bcb425a93ca191494677bbf0cb9e595d6d4b6272759"}
Jul 18 01:29:58.831604 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.831545    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:58.861443 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.861409    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zf8g\" (UniqueName: \"kubernetes.io/projected/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-kube-api-access-6zf8g\") pod \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") "
Jul 18 01:29:58.861653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.861458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-openshift-service-ca\") pod \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") "
Jul 18 01:29:58.861653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.861503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls\") pod \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\" (UID: \"77d5f5fe-174b-48f7-a7b2-277580f0fb2a\") "
Jul 18 01:29:58.861912 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.861880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77d5f5fe-174b-48f7-a7b2-277580f0fb2a" (UID: "77d5f5fe-174b-48f7-a7b2-277580f0fb2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:29:58.864109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.864085    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77d5f5fe-174b-48f7-a7b2-277580f0fb2a" (UID: "77d5f5fe-174b-48f7-a7b2-277580f0fb2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:29:58.864109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.864093    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-kube-api-access-6zf8g" (OuterVolumeSpecName: "kube-api-access-6zf8g") pod "77d5f5fe-174b-48f7-a7b2-277580f0fb2a" (UID: "77d5f5fe-174b-48f7-a7b2-277580f0fb2a"). InnerVolumeSpecName "kube-api-access-6zf8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:29:58.962423 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.962384    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zf8g\" (UniqueName: \"kubernetes.io/projected/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-kube-api-access-6zf8g\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:58.962423 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.962417    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:58.962423 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:58.962428    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d5f5fe-174b-48f7-a7b2-277580f0fb2a-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:29:59.756346 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.756262    2563 generic.go:356] "Generic (PLEG): container finished" podID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerID="ba3db1acdc4b7d7c44bdc1c6bf30358c4fb83ab362521fb697dc8d2f6592fd54" exitCode=6
Jul 18 01:29:59.756346 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.756299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" event={"ID":"bcabcdcc-1620-412a-a8e7-df6e5c157cd6","Type":"ContainerDied","Data":"ba3db1acdc4b7d7c44bdc1c6bf30358c4fb83ab362521fb697dc8d2f6592fd54"}
Jul 18 01:29:59.756852 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.756363    2563 scope.go:117] "RemoveContainer" containerID="0104f2584baa91ceb0e9ffca4425565aa456fc9dbed28e8997171ff3e250f8b2"
Jul 18 01:29:59.756852 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.756553    2563 scope.go:117] "RemoveContainer" containerID="ba3db1acdc4b7d7c44bdc1c6bf30358c4fb83ab362521fb697dc8d2f6592fd54"
Jul 18 01:29:59.756852 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:29:59.756775    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c_odh-ai-gateway-infra(bcabcdcc-1620-412a-a8e7-df6e5c157cd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6"
Jul 18 01:29:59.757836 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.757815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55" event={"ID":"77d5f5fe-174b-48f7-a7b2-277580f0fb2a","Type":"ContainerDied","Data":"429baeb59f9554baa5d506ac4c46e91df71d95f8f65c52ec58f569571572b295"}
Jul 18 01:29:59.757916 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.757837    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"
Jul 18 01:29:59.768580 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.768553    2563 scope.go:117] "RemoveContainer" containerID="a04eaa5bcf0b011725c09bcb425a93ca191494677bbf0cb9e595d6d4b6272759"
Jul 18 01:29:59.784785 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.784748    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"]
Jul 18 01:29:59.788512 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:29:59.788487    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0192d7-74d7947584-8dx55"]
Jul 18 01:30:00.148925 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.148822    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"]
Jul 18 01:30:00.149302 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.149285    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d5f5fe-174b-48f7-a7b2-277580f0fb2a" containerName="maas-api"
Jul 18 01:30:00.149394 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.149304    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d5f5fe-174b-48f7-a7b2-277580f0fb2a" containerName="maas-api"
Jul 18 01:30:00.149394 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.149389    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d5f5fe-174b-48f7-a7b2-277580f0fb2a" containerName="maas-api"
Jul 18 01:30:00.153716 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.153693    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2"]
Jul 18 01:30:00.153830 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.153808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"
Jul 18 01:30:00.157249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.157221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2"
Jul 18 01:30:00.159353 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.159150    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2"]
Jul 18 01:30:00.162400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.162373    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"]
Jul 18 01:30:00.168280 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.167707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"
Jul 18 01:30:00.175744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.175721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2"
Jul 18 01:30:00.179159 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.179128    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d5f5fe-174b-48f7-a7b2-277580f0fb2a" path="/var/lib/kubelet/pods/77d5f5fe-174b-48f7-a7b2-277580f0fb2a/volumes"
Jul 18 01:30:00.321985 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.321880    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"]
Jul 18 01:30:00.325211 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:30:00.325178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa855f7_9229_40bb_8ae5_de3aa82db99a.slice/crio-7328b47fa95454a3739275a0d863e0d24922bd72204dbc0b8decdebe9e8bd82d WatchSource:0}: Error finding container 7328b47fa95454a3739275a0d863e0d24922bd72204dbc0b8decdebe9e8bd82d: Status 404 returned error can't find the container with id 7328b47fa95454a3739275a0d863e0d24922bd72204dbc0b8decdebe9e8bd82d
Jul 18 01:30:00.353946 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.353917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2"]
Jul 18 01:30:00.356248 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:30:00.356223    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c42ac8_efeb_4713_84cc_0b6463c9f80f.slice/crio-82cedac25fbf1e171d1a4d8b931050c052bcb700e8365f9a6a8d4e9bc13a9b78 WatchSource:0}: Error finding container 82cedac25fbf1e171d1a4d8b931050c052bcb700e8365f9a6a8d4e9bc13a9b78: Status 404 returned error can't find the container with id 82cedac25fbf1e171d1a4d8b931050c052bcb700e8365f9a6a8d4e9bc13a9b78
Jul 18 01:30:00.766186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.766092    2563 scope.go:117] "RemoveContainer" containerID="ba3db1acdc4b7d7c44bdc1c6bf30358c4fb83ab362521fb697dc8d2f6592fd54"
Jul 18 01:30:00.766630 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:00.766340    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c_odh-ai-gateway-infra(bcabcdcc-1620-412a-a8e7-df6e5c157cd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6"
Jul 18 01:30:00.767632 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.767601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2" event={"ID":"29c42ac8-efeb-4713-84cc-0b6463c9f80f","Type":"ContainerStarted","Data":"a3839e3dd427f139292e2715cfd109ad36ab953990966d44915b0c492b8457b3"}
Jul 18 01:30:00.767744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.767655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2" event={"ID":"29c42ac8-efeb-4713-84cc-0b6463c9f80f","Type":"ContainerStarted","Data":"82cedac25fbf1e171d1a4d8b931050c052bcb700e8365f9a6a8d4e9bc13a9b78"}
Jul 18 01:30:00.777079 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.777002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62" event={"ID":"9aa855f7-9229-40bb-8ae5-de3aa82db99a","Type":"ContainerStarted","Data":"51e385c10cc4f3930a03f9a38f7339b4a0885f0aaad3c97cd81636cab6c4ea42"}
Jul 18 01:30:00.777251 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.777129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62" event={"ID":"9aa855f7-9229-40bb-8ae5-de3aa82db99a","Type":"ContainerStarted","Data":"7328b47fa95454a3739275a0d863e0d24922bd72204dbc0b8decdebe9e8bd82d"}
Jul 18 01:30:00.832367 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.832310    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62" podStartSLOduration=0.83229025 podStartE2EDuration="832.29025ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:00.832212181 +0000 UTC m=+2971.152189565" watchObservedRunningTime="2026-07-18 01:30:00.83229025 +0000 UTC m=+2971.152267633"
Jul 18 01:30:00.848518 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:00.848464    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2" podStartSLOduration=0.848446809 podStartE2EDuration="848.446809ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:00.847769533 +0000 UTC m=+2971.167746915" watchObservedRunningTime="2026-07-18 01:30:00.848446809 +0000 UTC m=+2971.168424191"
Jul 18 01:30:01.170225 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:01.170132    2563 scope.go:117] "RemoveContainer" containerID="3fba0ac7810561fe621fb92e0f0ad60811cc6237e36b03280344f9d8f99e7835"
Jul 18 01:30:01.786232 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:01.786134    2563 generic.go:356] "Generic (PLEG): container finished" podID="29c42ac8-efeb-4713-84cc-0b6463c9f80f" containerID="a3839e3dd427f139292e2715cfd109ad36ab953990966d44915b0c492b8457b3" exitCode=0
Jul 18 01:30:01.786232 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:01.786217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2" event={"ID":"29c42ac8-efeb-4713-84cc-0b6463c9f80f","Type":"ContainerDied","Data":"a3839e3dd427f139292e2715cfd109ad36ab953990966d44915b0c492b8457b3"}
Jul 18 01:30:01.788110 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:01.788084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" event={"ID":"0b3c6be1-36d1-451b-b9a5-125c92013471","Type":"ContainerStarted","Data":"f1c43cd02dfca149414ff9cb2b10c2e00bf508d1f70185db45857de67ed8b3fc"}
Jul 18 01:30:01.789561 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:01.789541    2563 generic.go:356] "Generic (PLEG): container finished" podID="9aa855f7-9229-40bb-8ae5-de3aa82db99a" containerID="51e385c10cc4f3930a03f9a38f7339b4a0885f0aaad3c97cd81636cab6c4ea42" exitCode=0
Jul 18 01:30:01.789661 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:01.789618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62" event={"ID":"9aa855f7-9229-40bb-8ae5-de3aa82db99a","Type":"ContainerDied","Data":"51e385c10cc4f3930a03f9a38f7339b4a0885f0aaad3c97cd81636cab6c4ea42"}
Jul 18 01:30:01.833870 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:01.833813    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" podStartSLOduration=19.833792766 podStartE2EDuration="19.833792766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:01.831726613 +0000 UTC m=+2972.151703995" watchObservedRunningTime="2026-07-18 01:30:01.833792766 +0000 UTC m=+2972.153770148"
Jul 18 01:30:02.530331 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:02.530291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:30:02.798159 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:02.798042    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerID="f1c43cd02dfca149414ff9cb2b10c2e00bf508d1f70185db45857de67ed8b3fc" exitCode=6
Jul 18 01:30:02.798159 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:02.798090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" event={"ID":"0b3c6be1-36d1-451b-b9a5-125c92013471","Type":"ContainerDied","Data":"f1c43cd02dfca149414ff9cb2b10c2e00bf508d1f70185db45857de67ed8b3fc"}
Jul 18 01:30:02.798159 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:02.798159    2563 scope.go:117] "RemoveContainer" containerID="3fba0ac7810561fe621fb92e0f0ad60811cc6237e36b03280344f9d8f99e7835"
Jul 18 01:30:02.798719 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:02.798471    2563 scope.go:117] "RemoveContainer" containerID="f1c43cd02dfca149414ff9cb2b10c2e00bf508d1f70185db45857de67ed8b3fc"
Jul 18 01:30:02.799178 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:02.798743    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv_odh-ai-gateway-infra(0b3c6be1-36d1-451b-b9a5-125c92013471)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471"
Jul 18 01:30:02.843886 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:02.843839    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"]
Jul 18 01:30:02.941135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:02.941109    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"
Jul 18 01:30:02.970302 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:02.970271    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2"
Jul 18 01:30:03.808014 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:03.807990    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"
Jul 18 01:30:03.808349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:03.808020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62" event={"ID":"9aa855f7-9229-40bb-8ae5-de3aa82db99a","Type":"ContainerDied","Data":"7328b47fa95454a3739275a0d863e0d24922bd72204dbc0b8decdebe9e8bd82d"}
Jul 18 01:30:03.808349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:03.808088    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7328b47fa95454a3739275a0d863e0d24922bd72204dbc0b8decdebe9e8bd82d"
Jul 18 01:30:03.809460 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:03.809438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2"
Jul 18 01:30:03.809591 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:03.809457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738970-2pkm2" event={"ID":"29c42ac8-efeb-4713-84cc-0b6463c9f80f","Type":"ContainerDied","Data":"82cedac25fbf1e171d1a4d8b931050c052bcb700e8365f9a6a8d4e9bc13a9b78"}
Jul 18 01:30:03.809591 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:03.809497    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82cedac25fbf1e171d1a4d8b931050c052bcb700e8365f9a6a8d4e9bc13a9b78"
Jul 18 01:30:03.946300 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:03.946275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"
Jul 18 01:30:04.008352 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:04.008304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c6be1-36d1-451b-b9a5-125c92013471-maas-api-service-ca\") pod \"0b3c6be1-36d1-451b-b9a5-125c92013471\" (UID: \"0b3c6be1-36d1-451b-b9a5-125c92013471\") "
Jul 18 01:30:04.008695 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:04.008666    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3c6be1-36d1-451b-b9a5-125c92013471-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b3c6be1-36d1-451b-b9a5-125c92013471" (UID: "0b3c6be1-36d1-451b-b9a5-125c92013471"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:30:04.109358 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:04.109269    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c6be1-36d1-451b-b9a5-125c92013471-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:30:04.816853 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:04.816815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv" event={"ID":"0b3c6be1-36d1-451b-b9a5-125c92013471","Type":"ContainerDied","Data":"8944269b5405361afd1fdeedca49a8d8a672909ab3a784d42312bf79dc0cf483"}
Jul 18 01:30:04.817332 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:04.816876    2563 scope.go:117] "RemoveContainer" containerID="f1c43cd02dfca149414ff9cb2b10c2e00bf508d1f70185db45857de67ed8b3fc"
Jul 18 01:30:04.817332 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:04.816824    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"
Jul 18 01:30:04.836222 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:04.836189    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"]
Jul 18 01:30:04.842692 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:04.842656    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-rwvvv"]
Jul 18 01:30:06.176131 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:06.176101    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" path="/var/lib/kubelet/pods/0b3c6be1-36d1-451b-b9a5-125c92013471/volumes"
Jul 18 01:30:13.169670 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.169628    2563 scope.go:117] "RemoveContainer" containerID="ba3db1acdc4b7d7c44bdc1c6bf30358c4fb83ab362521fb697dc8d2f6592fd54"
Jul 18 01:30:13.872377 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.872287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" event={"ID":"bcabcdcc-1620-412a-a8e7-df6e5c157cd6","Type":"ContainerStarted","Data":"79c946712427ec1f68c518bd512c8057066123270383e7ed8a63a057fb354e3d"}
Jul 18 01:30:13.901675 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.901627    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" podStartSLOduration=16.901611201 podStartE2EDuration="16.901611201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:13.89859984 +0000 UTC m=+2984.218577224" watchObservedRunningTime="2026-07-18 01:30:13.901611201 +0000 UTC m=+2984.221588584"
Jul 18 01:30:13.990499 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990460    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"]
Jul 18 01:30:13.990899 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990885    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:13.990950 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990902    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:13.990950 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990914    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa855f7-9229-40bb-8ae5-de3aa82db99a" containerName="cleanup"
Jul 18 01:30:13.990950 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990920    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa855f7-9229-40bb-8ae5-de3aa82db99a" containerName="cleanup"
Jul 18 01:30:13.990950 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990931    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c42ac8-efeb-4713-84cc-0b6463c9f80f" containerName="cleanup"
Jul 18 01:30:13.990950 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990937    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c42ac8-efeb-4713-84cc-0b6463c9f80f" containerName="cleanup"
Jul 18 01:30:13.990950 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990947    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:13.990950 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.990952    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:13.991186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.991017    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa855f7-9229-40bb-8ae5-de3aa82db99a" containerName="cleanup"
Jul 18 01:30:13.991186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.991025    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c42ac8-efeb-4713-84cc-0b6463c9f80f" containerName="cleanup"
Jul 18 01:30:13.991186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.991033    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:13.991186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.991040    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:13.994153 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:13.994134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"
Jul 18 01:30:14.001502 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.001469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"]
Jul 18 01:30:14.102178 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.102140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cbd8ed3-b649-4fda-93c3-00a4eb88dca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9\" (UID: \"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"
Jul 18 01:30:14.202820 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.202781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cbd8ed3-b649-4fda-93c3-00a4eb88dca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9\" (UID: \"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"
Jul 18 01:30:14.203449 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.203427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cbd8ed3-b649-4fda-93c3-00a4eb88dca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9\" (UID: \"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"
Jul 18 01:30:14.306048 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.306008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"
Jul 18 01:30:14.445799 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.445768    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"]
Jul 18 01:30:14.446915 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:30:14.446877    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbd8ed3_b649_4fda_93c3_00a4eb88dca9.slice/crio-e61f76fb8a0b4316d0b58bacd54bcfb847e1ea8976c1e18968fea4cf5ea8644b WatchSource:0}: Error finding container e61f76fb8a0b4316d0b58bacd54bcfb847e1ea8976c1e18968fea4cf5ea8644b: Status 404 returned error can't find the container with id e61f76fb8a0b4316d0b58bacd54bcfb847e1ea8976c1e18968fea4cf5ea8644b
Jul 18 01:30:14.881018 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.880981    2563 generic.go:356] "Generic (PLEG): container finished" podID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerID="79c946712427ec1f68c518bd512c8057066123270383e7ed8a63a057fb354e3d" exitCode=6
Jul 18 01:30:14.881250 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.881076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" event={"ID":"bcabcdcc-1620-412a-a8e7-df6e5c157cd6","Type":"ContainerDied","Data":"79c946712427ec1f68c518bd512c8057066123270383e7ed8a63a057fb354e3d"}
Jul 18 01:30:14.881250 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.881150    2563 scope.go:117] "RemoveContainer" containerID="ba3db1acdc4b7d7c44bdc1c6bf30358c4fb83ab362521fb697dc8d2f6592fd54"
Jul 18 01:30:14.881528 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.881485    2563 scope.go:117] "RemoveContainer" containerID="79c946712427ec1f68c518bd512c8057066123270383e7ed8a63a057fb354e3d"
Jul 18 01:30:14.881762 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:14.881736    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c_odh-ai-gateway-infra(bcabcdcc-1620-412a-a8e7-df6e5c157cd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6"
Jul 18 01:30:14.883180 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.883153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9" event={"ID":"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9","Type":"ContainerStarted","Data":"9595c6cc937d05a183f342684a101de6c6cf67e0b89306faecb5822665dded4b"}
Jul 18 01:30:14.883273 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.883203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9" event={"ID":"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9","Type":"ContainerStarted","Data":"e61f76fb8a0b4316d0b58bacd54bcfb847e1ea8976c1e18968fea4cf5ea8644b"}
Jul 18 01:30:14.909158 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.909120    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"]
Jul 18 01:30:14.910534 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:14.910490    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9" podStartSLOduration=1.9104752390000002 podStartE2EDuration="1.910475239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:14.907910439 +0000 UTC m=+2985.227887823" watchObservedRunningTime="2026-07-18 01:30:14.910475239 +0000 UTC m=+2985.230452647"
Jul 18 01:30:15.283100 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.283033    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"]
Jul 18 01:30:15.283546 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.283525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:15.283546 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.283544    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:15.283755 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.283657    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3c6be1-36d1-451b-b9a5-125c92013471" containerName="revoke-keys"
Jul 18 01:30:15.286764 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.286744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"
Jul 18 01:30:15.293282 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.293257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"]
Jul 18 01:30:15.412987 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.412946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3489937-ca03-4dc6-b379-55191e8ed984-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gqvlx\" (UID: \"d3489937-ca03-4dc6-b379-55191e8ed984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"
Jul 18 01:30:15.513597 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.513557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3489937-ca03-4dc6-b379-55191e8ed984-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gqvlx\" (UID: \"d3489937-ca03-4dc6-b379-55191e8ed984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"
Jul 18 01:30:15.514263 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.514244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3489937-ca03-4dc6-b379-55191e8ed984-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-gqvlx\" (UID: \"d3489937-ca03-4dc6-b379-55191e8ed984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"
Jul 18 01:30:15.599409 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.599366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"
Jul 18 01:30:15.737908 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.737883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"]
Jul 18 01:30:15.740194 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:30:15.740164    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3489937_ca03_4dc6_b379_55191e8ed984.slice/crio-66d8f60ea1ef74b9c88c13b80c2d40452d648b880c21c0541b188d64ed9deeb2 WatchSource:0}: Error finding container 66d8f60ea1ef74b9c88c13b80c2d40452d648b880c21c0541b188d64ed9deeb2: Status 404 returned error can't find the container with id 66d8f60ea1ef74b9c88c13b80c2d40452d648b880c21c0541b188d64ed9deeb2
Jul 18 01:30:15.893150 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.893119    2563 generic.go:356] "Generic (PLEG): container finished" podID="7cbd8ed3-b649-4fda-93c3-00a4eb88dca9" containerID="9595c6cc937d05a183f342684a101de6c6cf67e0b89306faecb5822665dded4b" exitCode=0
Jul 18 01:30:15.893249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.893166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9" event={"ID":"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9","Type":"ContainerDied","Data":"9595c6cc937d05a183f342684a101de6c6cf67e0b89306faecb5822665dded4b"}
Jul 18 01:30:15.894504 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:15.894483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx" event={"ID":"d3489937-ca03-4dc6-b379-55191e8ed984","Type":"ContainerStarted","Data":"66d8f60ea1ef74b9c88c13b80c2d40452d648b880c21c0541b188d64ed9deeb2"}
Jul 18 01:30:16.048398 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.048365    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"
Jul 18 01:30:16.123907 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.123857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcabcdcc-1620-412a-a8e7-df6e5c157cd6-maas-api-service-ca\") pod \"bcabcdcc-1620-412a-a8e7-df6e5c157cd6\" (UID: \"bcabcdcc-1620-412a-a8e7-df6e5c157cd6\") "
Jul 18 01:30:16.124284 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.124258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcabcdcc-1620-412a-a8e7-df6e5c157cd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bcabcdcc-1620-412a-a8e7-df6e5c157cd6" (UID: "bcabcdcc-1620-412a-a8e7-df6e5c157cd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:30:16.225401 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.225365    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcabcdcc-1620-412a-a8e7-df6e5c157cd6-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:30:16.902328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.902292    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3489937-ca03-4dc6-b379-55191e8ed984" containerID="e0e5890f760c674672e46ce8bcf6a156eaee32ea4160c03d66f1c7c34ceff47b" exitCode=6
Jul 18 01:30:16.902795 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.902381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx" event={"ID":"d3489937-ca03-4dc6-b379-55191e8ed984","Type":"ContainerDied","Data":"e0e5890f760c674672e46ce8bcf6a156eaee32ea4160c03d66f1c7c34ceff47b"}
Jul 18 01:30:16.902795 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.902568    2563 scope.go:117] "RemoveContainer" containerID="e0e5890f760c674672e46ce8bcf6a156eaee32ea4160c03d66f1c7c34ceff47b"
Jul 18 01:30:16.903888 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.903872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"
Jul 18 01:30:16.903993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.903904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c" event={"ID":"bcabcdcc-1620-412a-a8e7-df6e5c157cd6","Type":"ContainerDied","Data":"0662ce7dd1daa85a2e95d3f054f4f5d8fc5ae37a1c77af0bec4133798636e3c9"}
Jul 18 01:30:16.903993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.903948    2563 scope.go:117] "RemoveContainer" containerID="79c946712427ec1f68c518bd512c8057066123270383e7ed8a63a057fb354e3d"
Jul 18 01:30:16.934225 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.934191    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"]
Jul 18 01:30:16.937492 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:16.937466    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-f295c"]
Jul 18 01:30:17.045933 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.045909    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"
Jul 18 01:30:17.135966 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.135933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cbd8ed3-b649-4fda-93c3-00a4eb88dca9-maas-api-service-ca\") pod \"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9\" (UID: \"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9\") "
Jul 18 01:30:17.136355 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.136329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbd8ed3-b649-4fda-93c3-00a4eb88dca9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7cbd8ed3-b649-4fda-93c3-00a4eb88dca9" (UID: "7cbd8ed3-b649-4fda-93c3-00a4eb88dca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:30:17.236705 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.236671    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cbd8ed3-b649-4fda-93c3-00a4eb88dca9-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:30:17.912657 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.912559    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9"
Jul 18 01:30:17.912657 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.912602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9" event={"ID":"7cbd8ed3-b649-4fda-93c3-00a4eb88dca9","Type":"ContainerDied","Data":"e61f76fb8a0b4316d0b58bacd54bcfb847e1ea8976c1e18968fea4cf5ea8644b"}
Jul 18 01:30:17.913116 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.912665    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e61f76fb8a0b4316d0b58bacd54bcfb847e1ea8976c1e18968fea4cf5ea8644b"
Jul 18 01:30:17.914453 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.914427    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3489937-ca03-4dc6-b379-55191e8ed984" containerID="249cd3e843565fbc5005b4c5f47614beb7a8a35a685bee7f195c6c4156a68ebe" exitCode=6
Jul 18 01:30:17.914599 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.914509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx" event={"ID":"d3489937-ca03-4dc6-b379-55191e8ed984","Type":"ContainerDied","Data":"249cd3e843565fbc5005b4c5f47614beb7a8a35a685bee7f195c6c4156a68ebe"}
Jul 18 01:30:17.914599 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.914551    2563 scope.go:117] "RemoveContainer" containerID="e0e5890f760c674672e46ce8bcf6a156eaee32ea4160c03d66f1c7c34ceff47b"
Jul 18 01:30:17.914711 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:17.914663    2563 scope.go:117] "RemoveContainer" containerID="249cd3e843565fbc5005b4c5f47614beb7a8a35a685bee7f195c6c4156a68ebe"
Jul 18 01:30:17.914901 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:17.914880    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gqvlx_odh-ai-gateway-infra(d3489937-ca03-4dc6-b379-55191e8ed984)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx" podUID="d3489937-ca03-4dc6-b379-55191e8ed984"
Jul 18 01:30:18.176801 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:18.176691    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" path="/var/lib/kubelet/pods/bcabcdcc-1620-412a-a8e7-df6e5c157cd6/volumes"
Jul 18 01:30:18.923295 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:18.923261    2563 scope.go:117] "RemoveContainer" containerID="249cd3e843565fbc5005b4c5f47614beb7a8a35a685bee7f195c6c4156a68ebe"
Jul 18 01:30:18.923742 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:18.923509    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gqvlx_odh-ai-gateway-infra(d3489937-ca03-4dc6-b379-55191e8ed984)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx" podUID="d3489937-ca03-4dc6-b379-55191e8ed984"
Jul 18 01:30:24.556977 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.556911    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"]
Jul 18 01:30:24.557492 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.557221    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw" podUID="c8b235a3-32b1-40cc-be09-bc20df7f2b6d" containerName="maas-api" containerID="cri-o://393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666" gracePeriod=30
Jul 18 01:30:24.587840 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.587793    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"]
Jul 18 01:30:24.590631 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.590601    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-232dbd-29738970-8tl62"]
Jul 18 01:30:24.820276 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.820212    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:30:24.959126 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.959088    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8b235a3-32b1-40cc-be09-bc20df7f2b6d" containerID="393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666" exitCode=0
Jul 18 01:30:24.959278 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.959135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw" event={"ID":"c8b235a3-32b1-40cc-be09-bc20df7f2b6d","Type":"ContainerDied","Data":"393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666"}
Jul 18 01:30:24.959278 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.959181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw" event={"ID":"c8b235a3-32b1-40cc-be09-bc20df7f2b6d","Type":"ContainerDied","Data":"467e0b5e502d1f31f66c72dc11ad909b21447e9704596cc9f725592df599b84e"}
Jul 18 01:30:24.959278 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.959192    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"
Jul 18 01:30:24.959278 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.959207    2563 scope.go:117] "RemoveContainer" containerID="393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666"
Jul 18 01:30:24.968024 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.968002    2563 scope.go:117] "RemoveContainer" containerID="393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666"
Jul 18 01:30:24.968361 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:24.968340    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666\": container with ID starting with 393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666 not found: ID does not exist" containerID="393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666"
Jul 18 01:30:24.968426 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:24.968372    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666"} err="failed to get container status \"393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666\": rpc error: code = NotFound desc = could not find container \"393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666\": container with ID starting with 393dbc97de753ff3d0c311f18a2000f853dcc7aa10f2b7c8247bb0df067d0666 not found: ID does not exist"
Jul 18 01:30:25.007533 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.007503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls\") pod \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") "
Jul 18 01:30:25.007698 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.007564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qndk5\" (UniqueName: \"kubernetes.io/projected/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-kube-api-access-qndk5\") pod \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") "
Jul 18 01:30:25.007698 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.007609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-openshift-service-ca\") pod \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\" (UID: \"c8b235a3-32b1-40cc-be09-bc20df7f2b6d\") "
Jul 18 01:30:25.008024 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.007987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8b235a3-32b1-40cc-be09-bc20df7f2b6d" (UID: "c8b235a3-32b1-40cc-be09-bc20df7f2b6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:30:25.009610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.009585    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-kube-api-access-qndk5" (OuterVolumeSpecName: "kube-api-access-qndk5") pod "c8b235a3-32b1-40cc-be09-bc20df7f2b6d" (UID: "c8b235a3-32b1-40cc-be09-bc20df7f2b6d"). InnerVolumeSpecName "kube-api-access-qndk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:30:25.009675 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.009618    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8b235a3-32b1-40cc-be09-bc20df7f2b6d" (UID: "c8b235a3-32b1-40cc-be09-bc20df7f2b6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:30:25.108480 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.108377    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qndk5\" (UniqueName: \"kubernetes.io/projected/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-kube-api-access-qndk5\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:30:25.108480 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.108417    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:30:25.108480 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.108432    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8b235a3-32b1-40cc-be09-bc20df7f2b6d-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:30:25.282580 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.282544    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"]
Jul 18 01:30:25.285127 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:25.285100    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-232dbd-86d8f8cbdf-q8xcw"]
Jul 18 01:30:26.183684 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:26.183629    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa855f7-9229-40bb-8ae5-de3aa82db99a" path="/var/lib/kubelet/pods/9aa855f7-9229-40bb-8ae5-de3aa82db99a/volumes"
Jul 18 01:30:26.184095 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:26.184014    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8b235a3-32b1-40cc-be09-bc20df7f2b6d" path="/var/lib/kubelet/pods/c8b235a3-32b1-40cc-be09-bc20df7f2b6d/volumes"
Jul 18 01:30:27.538044 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:27.538001    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:30:33.170314 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:33.170274    2563 scope.go:117] "RemoveContainer" containerID="249cd3e843565fbc5005b4c5f47614beb7a8a35a685bee7f195c6c4156a68ebe"
Jul 18 01:30:34.017282 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:34.017239    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3489937-ca03-4dc6-b379-55191e8ed984" containerID="b3fa8ae2c6c65aa213097748dd491718e67172785e7db132eee49166bcf0e746" exitCode=6
Jul 18 01:30:34.017476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:34.017312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx" event={"ID":"d3489937-ca03-4dc6-b379-55191e8ed984","Type":"ContainerDied","Data":"b3fa8ae2c6c65aa213097748dd491718e67172785e7db132eee49166bcf0e746"}
Jul 18 01:30:34.017476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:34.017373    2563 scope.go:117] "RemoveContainer" containerID="249cd3e843565fbc5005b4c5f47614beb7a8a35a685bee7f195c6c4156a68ebe"
Jul 18 01:30:34.017734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:34.017718    2563 scope.go:117] "RemoveContainer" containerID="b3fa8ae2c6c65aa213097748dd491718e67172785e7db132eee49166bcf0e746"
Jul 18 01:30:34.017950 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:34.017928    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-gqvlx_odh-ai-gateway-infra(d3489937-ca03-4dc6-b379-55191e8ed984)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx" podUID="d3489937-ca03-4dc6-b379-55191e8ed984"
Jul 18 01:30:35.043333 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:35.043295    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"]
Jul 18 01:30:35.188691 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:35.188656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"
Jul 18 01:30:35.287928 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:35.287886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3489937-ca03-4dc6-b379-55191e8ed984-maas-api-service-ca\") pod \"d3489937-ca03-4dc6-b379-55191e8ed984\" (UID: \"d3489937-ca03-4dc6-b379-55191e8ed984\") "
Jul 18 01:30:35.288278 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:35.288254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3489937-ca03-4dc6-b379-55191e8ed984-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3489937-ca03-4dc6-b379-55191e8ed984" (UID: "d3489937-ca03-4dc6-b379-55191e8ed984"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:30:35.388516 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:35.388426    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3489937-ca03-4dc6-b379-55191e8ed984-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:30:36.030847 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:36.030813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"
Jul 18 01:30:36.031033 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:36.030855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx" event={"ID":"d3489937-ca03-4dc6-b379-55191e8ed984","Type":"ContainerDied","Data":"66d8f60ea1ef74b9c88c13b80c2d40452d648b880c21c0541b188d64ed9deeb2"}
Jul 18 01:30:36.031033 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:36.030921    2563 scope.go:117] "RemoveContainer" containerID="b3fa8ae2c6c65aa213097748dd491718e67172785e7db132eee49166bcf0e746"
Jul 18 01:30:36.054616 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:36.054582    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"]
Jul 18 01:30:36.060856 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:36.060827    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-gqvlx"]
Jul 18 01:30:36.175629 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:36.175597    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" path="/var/lib/kubelet/pods/d3489937-ca03-4dc6-b379-55191e8ed984/volumes"
Jul 18 01:30:42.381592 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.381551    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"]
Jul 18 01:30:42.382013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.381953    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:42.382013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.381965    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:42.382013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.381974    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:42.382013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.381980    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:42.382013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.381989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:42.382013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.381995    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:42.382013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382007    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbd8ed3-b649-4fda-93c3-00a4eb88dca9" containerName="revoke-keys"
Jul 18 01:30:42.382013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382012    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbd8ed3-b649-4fda-93c3-00a4eb88dca9" containerName="revoke-keys"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382022    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b235a3-32b1-40cc-be09-bc20df7f2b6d" containerName="maas-api"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382028    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b235a3-32b1-40cc-be09-bc20df7f2b6d" containerName="maas-api"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382045    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382050    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382128    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382137    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382143    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382150    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbd8ed3-b649-4fda-93c3-00a4eb88dca9" containerName="revoke-keys"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382156    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:42.382342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.382161    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b235a3-32b1-40cc-be09-bc20df7f2b6d" containerName="maas-api"
Jul 18 01:30:42.385439 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.385419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"
Jul 18 01:30:42.387989 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.387962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:30:42.393392 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.393361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"]
Jul 18 01:30:42.447953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.447911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007d58a2-871d-4ebd-9fdf-b44f7872e165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h\" (UID: \"007d58a2-871d-4ebd-9fdf-b44f7872e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"
Jul 18 01:30:42.548584 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.548534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007d58a2-871d-4ebd-9fdf-b44f7872e165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h\" (UID: \"007d58a2-871d-4ebd-9fdf-b44f7872e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"
Jul 18 01:30:42.549250 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.549229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007d58a2-871d-4ebd-9fdf-b44f7872e165-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h\" (UID: \"007d58a2-871d-4ebd-9fdf-b44f7872e165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"
Jul 18 01:30:42.698367 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.698333    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"
Jul 18 01:30:42.842777 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:42.842742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"]
Jul 18 01:30:42.844534 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:30:42.844505    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007d58a2_871d_4ebd_9fdf_b44f7872e165.slice/crio-17289f597abe49fc7c3a99f7a8bb8b9ad9b518670a2764ad946a2615e33ddea4 WatchSource:0}: Error finding container 17289f597abe49fc7c3a99f7a8bb8b9ad9b518670a2764ad946a2615e33ddea4: Status 404 returned error can't find the container with id 17289f597abe49fc7c3a99f7a8bb8b9ad9b518670a2764ad946a2615e33ddea4
Jul 18 01:30:43.074939 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.074846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h" event={"ID":"007d58a2-871d-4ebd-9fdf-b44f7872e165","Type":"ContainerStarted","Data":"17289f597abe49fc7c3a99f7a8bb8b9ad9b518670a2764ad946a2615e33ddea4"}
Jul 18 01:30:43.906493 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.906457    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"]
Jul 18 01:30:43.906929 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.906913    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:43.906979 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.906932    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:43.906979 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.906946    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:43.906979 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.906951    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:43.907137 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.907041    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3489937-ca03-4dc6-b379-55191e8ed984" containerName="revoke-keys"
Jul 18 01:30:43.907137 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.907069    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcabcdcc-1620-412a-a8e7-df6e5c157cd6" containerName="revoke-keys"
Jul 18 01:30:43.910432 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.910408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:43.913043 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.913014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ad1b67-serving-cert\""
Jul 18 01:30:43.921180 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.921153    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"]
Jul 18 01:30:43.961963 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.961933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77620092-1cbc-486a-a296-5a7d54ec1bfa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:43.962155 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.962038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhlb\" (UniqueName: \"kubernetes.io/projected/77620092-1cbc-486a-a296-5a7d54ec1bfa-kube-api-access-cjhlb\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:43.962155 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:43.962107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.063109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.063066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhlb\" (UniqueName: \"kubernetes.io/projected/77620092-1cbc-486a-a296-5a7d54ec1bfa-kube-api-access-cjhlb\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.063109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.063109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.063343 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.063162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77620092-1cbc-486a-a296-5a7d54ec1bfa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.063343 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:44.063294    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-serving-cert: secret "maas-api-e2e-shared-a-ad1b67-serving-cert" not found
Jul 18 01:30:44.063424 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:44.063358    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls podName:77620092-1cbc-486a-a296-5a7d54ec1bfa nodeName:}" failed. No retries permitted until 2026-07-18 01:30:44.563337731 +0000 UTC m=+3014.883315091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls") pod "maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6" (UID: "77620092-1cbc-486a-a296-5a7d54ec1bfa") : secret "maas-api-e2e-shared-a-ad1b67-serving-cert" not found
Jul 18 01:30:44.063885 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.063866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77620092-1cbc-486a-a296-5a7d54ec1bfa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.075130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.075098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhlb\" (UniqueName: \"kubernetes.io/projected/77620092-1cbc-486a-a296-5a7d54ec1bfa-kube-api-access-cjhlb\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.085845 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.085817    2563 generic.go:356] "Generic (PLEG): container finished" podID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerID="fe3cd672c16ff8dd8c26bb5214524d4667dbb258b91e33cb696388ef093138d6" exitCode=6
Jul 18 01:30:44.086001 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.085902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h" event={"ID":"007d58a2-871d-4ebd-9fdf-b44f7872e165","Type":"ContainerDied","Data":"fe3cd672c16ff8dd8c26bb5214524d4667dbb258b91e33cb696388ef093138d6"}
Jul 18 01:30:44.086142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.086123    2563 scope.go:117] "RemoveContainer" containerID="fe3cd672c16ff8dd8c26bb5214524d4667dbb258b91e33cb696388ef093138d6"
Jul 18 01:30:44.574830 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.574714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.577196 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.577170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls\") pod \"maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.826311 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.826231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:44.966523 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:44.966494    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"]
Jul 18 01:30:44.967870 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:30:44.967844    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77620092_1cbc_486a_a296_5a7d54ec1bfa.slice/crio-44ec0f0896bacb22d892ea06aea32afbc6c549e9cfa447d98378cf5da4d7759b WatchSource:0}: Error finding container 44ec0f0896bacb22d892ea06aea32afbc6c549e9cfa447d98378cf5da4d7759b: Status 404 returned error can't find the container with id 44ec0f0896bacb22d892ea06aea32afbc6c549e9cfa447d98378cf5da4d7759b
Jul 18 01:30:45.094861 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:45.094764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6" event={"ID":"77620092-1cbc-486a-a296-5a7d54ec1bfa","Type":"ContainerStarted","Data":"44ec0f0896bacb22d892ea06aea32afbc6c549e9cfa447d98378cf5da4d7759b"}
Jul 18 01:30:45.096678 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:45.096650    2563 generic.go:356] "Generic (PLEG): container finished" podID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerID="c0c96cbc2ca61b7bcc1f66a8644cb2b3de8af8b60ca52bde3c9f154c715773fa" exitCode=6
Jul 18 01:30:45.096792 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:45.096729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h" event={"ID":"007d58a2-871d-4ebd-9fdf-b44f7872e165","Type":"ContainerDied","Data":"c0c96cbc2ca61b7bcc1f66a8644cb2b3de8af8b60ca52bde3c9f154c715773fa"}
Jul 18 01:30:45.096792 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:45.096783    2563 scope.go:117] "RemoveContainer" containerID="fe3cd672c16ff8dd8c26bb5214524d4667dbb258b91e33cb696388ef093138d6"
Jul 18 01:30:45.097028 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:45.097011    2563 scope.go:117] "RemoveContainer" containerID="c0c96cbc2ca61b7bcc1f66a8644cb2b3de8af8b60ca52bde3c9f154c715773fa"
Jul 18 01:30:45.097271 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:45.097252    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h_odh-ai-gateway-infra(007d58a2-871d-4ebd-9fdf-b44f7872e165)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165"
Jul 18 01:30:46.104612 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:46.104564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6" event={"ID":"77620092-1cbc-486a-a296-5a7d54ec1bfa","Type":"ContainerStarted","Data":"4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a"}
Jul 18 01:30:46.105092 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:46.104646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:46.106397 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:46.106376    2563 scope.go:117] "RemoveContainer" containerID="c0c96cbc2ca61b7bcc1f66a8644cb2b3de8af8b60ca52bde3c9f154c715773fa"
Jul 18 01:30:46.106567 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:46.106550    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h_odh-ai-gateway-infra(007d58a2-871d-4ebd-9fdf-b44f7872e165)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165"
Jul 18 01:30:46.121231 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:46.121180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6" podStartSLOduration=2.705055786 podStartE2EDuration="3.121162342s" totalImagesPullingTime="416.106556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:46.119811535 +0000 UTC m=+3016.439788919" watchObservedRunningTime="2026-07-18 01:30:46.121162342 +0000 UTC m=+3016.441139724"
Jul 18 01:30:48.939571 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:48.939532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:30:51.982887 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:51.982836    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"]
Jul 18 01:30:51.986769 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:51.986741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:51.989526 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:51.989505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-0f7fdc-serving-cert\""
Jul 18 01:30:52.000016 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:51.999935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"]
Jul 18 01:30:52.041261 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.041222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ff4b1-b150-4ca9-915b-6166c729bad1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.041453 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.041307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494fz\" (UniqueName: \"kubernetes.io/projected/2a8ff4b1-b150-4ca9-915b-6166c729bad1-kube-api-access-494fz\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.041453 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.041378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.118423 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.118382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:30:52.141912 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.141875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ff4b1-b150-4ca9-915b-6166c729bad1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.142106 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.141957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494fz\" (UniqueName: \"kubernetes.io/projected/2a8ff4b1-b150-4ca9-915b-6166c729bad1-kube-api-access-494fz\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.142106 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.142025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.142233 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:52.142167    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-serving-cert: secret "maas-api-e2e-shared-b-0f7fdc-serving-cert" not found
Jul 18 01:30:52.142233 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:30:52.142228    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls podName:2a8ff4b1-b150-4ca9-915b-6166c729bad1 nodeName:}" failed. No retries permitted until 2026-07-18 01:30:52.64220668 +0000 UTC m=+3022.962184046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls") pod "maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h" (UID: "2a8ff4b1-b150-4ca9-915b-6166c729bad1") : secret "maas-api-e2e-shared-b-0f7fdc-serving-cert" not found
Jul 18 01:30:52.142762 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.142732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ff4b1-b150-4ca9-915b-6166c729bad1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.151961 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.151924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-494fz\" (UniqueName: \"kubernetes.io/projected/2a8ff4b1-b150-4ca9-915b-6166c729bad1-kube-api-access-494fz\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.646211 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.646171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.648711 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.648685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls\") pod \"maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:52.897931 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:52.897834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:53.046975 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:53.046941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"]
Jul 18 01:30:53.048437 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:30:53.048411    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8ff4b1_b150_4ca9_915b_6166c729bad1.slice/crio-ecfb641e083eec8c7f7c7e6d8ac7d349bf571386924f2511996eadd4e4a9ccba WatchSource:0}: Error finding container ecfb641e083eec8c7f7c7e6d8ac7d349bf571386924f2511996eadd4e4a9ccba: Status 404 returned error can't find the container with id ecfb641e083eec8c7f7c7e6d8ac7d349bf571386924f2511996eadd4e4a9ccba
Jul 18 01:30:53.151841 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:53.151740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h" event={"ID":"2a8ff4b1-b150-4ca9-915b-6166c729bad1","Type":"ContainerStarted","Data":"ecfb641e083eec8c7f7c7e6d8ac7d349bf571386924f2511996eadd4e4a9ccba"}
Jul 18 01:30:54.159248 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:54.159202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h" event={"ID":"2a8ff4b1-b150-4ca9-915b-6166c729bad1","Type":"ContainerStarted","Data":"8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173"}
Jul 18 01:30:54.159686 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:54.159435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:30:54.175918 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:54.175868    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h" podStartSLOduration=2.685914393 podStartE2EDuration="3.175852373s" totalImagesPullingTime="489.93798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:54.174420446 +0000 UTC m=+3024.494397828" watchObservedRunningTime="2026-07-18 01:30:54.175852373 +0000 UTC m=+3024.495829757"
Jul 18 01:30:56.840701 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:56.840652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:30:59.169851 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:30:59.169813    2563 scope.go:117] "RemoveContainer" containerID="c0c96cbc2ca61b7bcc1f66a8644cb2b3de8af8b60ca52bde3c9f154c715773fa"
Jul 18 01:31:00.176942 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:00.176864    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:31:00.199280 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:00.199249    2563 generic.go:356] "Generic (PLEG): container finished" podID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerID="ed5cfac3b07b57a75dce8e912cd5dda58a5918b94c63995db370123164a0bb0b" exitCode=6
Jul 18 01:31:00.199486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:00.199282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h" event={"ID":"007d58a2-871d-4ebd-9fdf-b44f7872e165","Type":"ContainerDied","Data":"ed5cfac3b07b57a75dce8e912cd5dda58a5918b94c63995db370123164a0bb0b"}
Jul 18 01:31:00.199486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:00.199351    2563 scope.go:117] "RemoveContainer" containerID="c0c96cbc2ca61b7bcc1f66a8644cb2b3de8af8b60ca52bde3c9f154c715773fa"
Jul 18 01:31:00.199827 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:00.199810    2563 scope.go:117] "RemoveContainer" containerID="ed5cfac3b07b57a75dce8e912cd5dda58a5918b94c63995db370123164a0bb0b"
Jul 18 01:31:00.200085 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:00.200039    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h_odh-ai-gateway-infra(007d58a2-871d-4ebd-9fdf-b44f7872e165)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165"
Jul 18 01:31:01.225659 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:01.225623    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"]
Jul 18 01:31:01.372996 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:01.372965    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"
Jul 18 01:31:01.431825 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:01.431790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007d58a2-871d-4ebd-9fdf-b44f7872e165-maas-api-service-ca\") pod \"007d58a2-871d-4ebd-9fdf-b44f7872e165\" (UID: \"007d58a2-871d-4ebd-9fdf-b44f7872e165\") "
Jul 18 01:31:01.432177 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:01.432151    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007d58a2-871d-4ebd-9fdf-b44f7872e165-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "007d58a2-871d-4ebd-9fdf-b44f7872e165" (UID: "007d58a2-871d-4ebd-9fdf-b44f7872e165"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:31:01.533313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:01.533210    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/007d58a2-871d-4ebd-9fdf-b44f7872e165-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:31:02.214311 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:02.214279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h" event={"ID":"007d58a2-871d-4ebd-9fdf-b44f7872e165","Type":"ContainerDied","Data":"17289f597abe49fc7c3a99f7a8bb8b9ad9b518670a2764ad946a2615e33ddea4"}
Jul 18 01:31:02.214531 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:02.214341    2563 scope.go:117] "RemoveContainer" containerID="ed5cfac3b07b57a75dce8e912cd5dda58a5918b94c63995db370123164a0bb0b"
Jul 18 01:31:02.214531 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:02.214283    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"
Jul 18 01:31:02.232804 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:02.232774    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"]
Jul 18 01:31:02.238344 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:02.238320    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-xp25h"]
Jul 18 01:31:04.176004 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:04.175967    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" path="/var/lib/kubelet/pods/007d58a2-871d-4ebd-9fdf-b44f7872e165/volumes"
Jul 18 01:31:11.346492 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:11.346452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:31:15.340557 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.340517    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"]
Jul 18 01:31:15.340985 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.340924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:15.340985 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.340937    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:15.340985 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.340953    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:15.340985 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.340959    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:15.341147 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.341036    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:15.341147 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.341045    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:15.344390 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.344371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"
Jul 18 01:31:15.347230 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.347206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:31:15.354038 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.354013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"]
Jul 18 01:31:15.456807 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.456770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912333d-c04b-4eff-9420-53f29985813c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-6h6l6\" (UID: \"9912333d-c04b-4eff-9420-53f29985813c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"
Jul 18 01:31:15.558130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.558091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912333d-c04b-4eff-9420-53f29985813c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-6h6l6\" (UID: \"9912333d-c04b-4eff-9420-53f29985813c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"
Jul 18 01:31:15.558751 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.558728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912333d-c04b-4eff-9420-53f29985813c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-6h6l6\" (UID: \"9912333d-c04b-4eff-9420-53f29985813c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"
Jul 18 01:31:15.656228 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.656137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"
Jul 18 01:31:15.800500 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:15.800473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"]
Jul 18 01:31:15.801898 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:31:15.801870    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9912333d_c04b_4eff_9420_53f29985813c.slice/crio-3398c90aa6b5daa617461bc725d2f62944bcfaef14b100216caf64a86c22598e WatchSource:0}: Error finding container 3398c90aa6b5daa617461bc725d2f62944bcfaef14b100216caf64a86c22598e: Status 404 returned error can't find the container with id 3398c90aa6b5daa617461bc725d2f62944bcfaef14b100216caf64a86c22598e
Jul 18 01:31:16.299294 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:16.299193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" event={"ID":"9912333d-c04b-4eff-9420-53f29985813c","Type":"ContainerStarted","Data":"0fb3a2ca7ab50b242f914b0598f05654cb894a52f3c6b8662e35bc5ba89b95ef"}
Jul 18 01:31:16.299294 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:16.299261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" event={"ID":"9912333d-c04b-4eff-9420-53f29985813c","Type":"ContainerStarted","Data":"3398c90aa6b5daa617461bc725d2f62944bcfaef14b100216caf64a86c22598e"}
Jul 18 01:31:16.315467 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:16.315413    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" podStartSLOduration=1.315395809 podStartE2EDuration="1.315395809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:31:16.31334998 +0000 UTC m=+3046.633327363" watchObservedRunningTime="2026-07-18 01:31:16.315395809 +0000 UTC m=+3046.635373215"
Jul 18 01:31:17.306136 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:17.306100    2563 generic.go:356] "Generic (PLEG): container finished" podID="9912333d-c04b-4eff-9420-53f29985813c" containerID="0fb3a2ca7ab50b242f914b0598f05654cb894a52f3c6b8662e35bc5ba89b95ef" exitCode=6
Jul 18 01:31:17.306540 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:17.306184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" event={"ID":"9912333d-c04b-4eff-9420-53f29985813c","Type":"ContainerDied","Data":"0fb3a2ca7ab50b242f914b0598f05654cb894a52f3c6b8662e35bc5ba89b95ef"}
Jul 18 01:31:17.306540 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:17.306366    2563 scope.go:117] "RemoveContainer" containerID="0fb3a2ca7ab50b242f914b0598f05654cb894a52f3c6b8662e35bc5ba89b95ef"
Jul 18 01:31:18.313914 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:18.313877    2563 generic.go:356] "Generic (PLEG): container finished" podID="9912333d-c04b-4eff-9420-53f29985813c" containerID="c7f10cbfde57bee39c36cb65a4a40af917eb2839dd2518d635b4b5094ca85a4f" exitCode=6
Jul 18 01:31:18.314389 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:18.313970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" event={"ID":"9912333d-c04b-4eff-9420-53f29985813c","Type":"ContainerDied","Data":"c7f10cbfde57bee39c36cb65a4a40af917eb2839dd2518d635b4b5094ca85a4f"}
Jul 18 01:31:18.314389 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:18.314030    2563 scope.go:117] "RemoveContainer" containerID="0fb3a2ca7ab50b242f914b0598f05654cb894a52f3c6b8662e35bc5ba89b95ef"
Jul 18 01:31:18.314389 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:18.314272    2563 scope.go:117] "RemoveContainer" containerID="c7f10cbfde57bee39c36cb65a4a40af917eb2839dd2518d635b4b5094ca85a4f"
Jul 18 01:31:18.314534 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:18.314510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-6h6l6_odh-ai-gateway-infra(9912333d-c04b-4eff-9420-53f29985813c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" podUID="9912333d-c04b-4eff-9420-53f29985813c"
Jul 18 01:31:19.278692 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.278656    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"]
Jul 18 01:31:19.279197 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.279177    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:19.279197 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.279198    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:19.279386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.279310    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="007d58a2-871d-4ebd-9fdf-b44f7872e165" containerName="revoke-keys"
Jul 18 01:31:19.283857 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.283834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"
Jul 18 01:31:19.291591 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.291560    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"]
Jul 18 01:31:19.322603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.322577    2563 scope.go:117] "RemoveContainer" containerID="c7f10cbfde57bee39c36cb65a4a40af917eb2839dd2518d635b4b5094ca85a4f"
Jul 18 01:31:19.322989 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:19.322776    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-6h6l6_odh-ai-gateway-infra(9912333d-c04b-4eff-9420-53f29985813c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" podUID="9912333d-c04b-4eff-9420-53f29985813c"
Jul 18 01:31:19.393112 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.393069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddba1201-18be-4333-b963-3a6f9abf179a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl\" (UID: \"ddba1201-18be-4333-b963-3a6f9abf179a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"
Jul 18 01:31:19.493564 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.493524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddba1201-18be-4333-b963-3a6f9abf179a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl\" (UID: \"ddba1201-18be-4333-b963-3a6f9abf179a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"
Jul 18 01:31:19.494247 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.494220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddba1201-18be-4333-b963-3a6f9abf179a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl\" (UID: \"ddba1201-18be-4333-b963-3a6f9abf179a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"
Jul 18 01:31:19.597942 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.597851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"
Jul 18 01:31:19.740446 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:19.740412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"]
Jul 18 01:31:19.742307 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:31:19.742278    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddba1201_18be_4333_b963_3a6f9abf179a.slice/crio-106c2ced4f3d17bdb9ee9caf1663f9c948ade1b5dfd1706bb4680e45a8b26d76 WatchSource:0}: Error finding container 106c2ced4f3d17bdb9ee9caf1663f9c948ade1b5dfd1706bb4680e45a8b26d76: Status 404 returned error can't find the container with id 106c2ced4f3d17bdb9ee9caf1663f9c948ade1b5dfd1706bb4680e45a8b26d76
Jul 18 01:31:20.330100 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:20.329988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" event={"ID":"ddba1201-18be-4333-b963-3a6f9abf179a","Type":"ContainerStarted","Data":"4fd20254ff28484586b491bd3d104239fba2b5b8d08c1ef8ef6984144e740c1e"}
Jul 18 01:31:20.330100 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:20.330049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" event={"ID":"ddba1201-18be-4333-b963-3a6f9abf179a","Type":"ContainerStarted","Data":"106c2ced4f3d17bdb9ee9caf1663f9c948ade1b5dfd1706bb4680e45a8b26d76"}
Jul 18 01:31:20.349175 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:20.349125    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" podStartSLOduration=1.34910661 podStartE2EDuration="1.34910661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:31:20.346547415 +0000 UTC m=+3050.666524799" watchObservedRunningTime="2026-07-18 01:31:20.34910661 +0000 UTC m=+3050.669083994"
Jul 18 01:31:21.337930 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:21.337894    2563 generic.go:356] "Generic (PLEG): container finished" podID="ddba1201-18be-4333-b963-3a6f9abf179a" containerID="4fd20254ff28484586b491bd3d104239fba2b5b8d08c1ef8ef6984144e740c1e" exitCode=6
Jul 18 01:31:21.338364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:21.337958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" event={"ID":"ddba1201-18be-4333-b963-3a6f9abf179a","Type":"ContainerDied","Data":"4fd20254ff28484586b491bd3d104239fba2b5b8d08c1ef8ef6984144e740c1e"}
Jul 18 01:31:21.338364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:21.338142    2563 scope.go:117] "RemoveContainer" containerID="4fd20254ff28484586b491bd3d104239fba2b5b8d08c1ef8ef6984144e740c1e"
Jul 18 01:31:22.345542 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:22.345427    2563 generic.go:356] "Generic (PLEG): container finished" podID="ddba1201-18be-4333-b963-3a6f9abf179a" containerID="75f56e1dc20bfd2403ff7a0e5ee608ebbd6f9e39772a43ebd8fa5a22ed666464" exitCode=6
Jul 18 01:31:22.345542 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:22.345476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" event={"ID":"ddba1201-18be-4333-b963-3a6f9abf179a","Type":"ContainerDied","Data":"75f56e1dc20bfd2403ff7a0e5ee608ebbd6f9e39772a43ebd8fa5a22ed666464"}
Jul 18 01:31:22.345542 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:22.345531    2563 scope.go:117] "RemoveContainer" containerID="4fd20254ff28484586b491bd3d104239fba2b5b8d08c1ef8ef6984144e740c1e"
Jul 18 01:31:22.346025 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:22.345733    2563 scope.go:117] "RemoveContainer" containerID="75f56e1dc20bfd2403ff7a0e5ee608ebbd6f9e39772a43ebd8fa5a22ed666464"
Jul 18 01:31:22.346025 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:22.345953    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl_odh-ai-gateway-infra(ddba1201-18be-4333-b963-3a6f9abf179a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" podUID="ddba1201-18be-4333-b963-3a6f9abf179a"
Jul 18 01:31:23.353418 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:23.353389    2563 scope.go:117] "RemoveContainer" containerID="75f56e1dc20bfd2403ff7a0e5ee608ebbd6f9e39772a43ebd8fa5a22ed666464"
Jul 18 01:31:23.353809 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:23.353604    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl_odh-ai-gateway-infra(ddba1201-18be-4333-b963-3a6f9abf179a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" podUID="ddba1201-18be-4333-b963-3a6f9abf179a"
Jul 18 01:31:32.425347 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:32.425308    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"]
Jul 18 01:31:32.430293 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:32.430271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"
Jul 18 01:31:32.435245 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:32.435219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"]
Jul 18 01:31:32.495361 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:32.495317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf98ba4-62e7-4cce-beea-2fd4a0b87e4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp\" (UID: \"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"
Jul 18 01:31:32.596220 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:32.596179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf98ba4-62e7-4cce-beea-2fd4a0b87e4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp\" (UID: \"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"
Jul 18 01:31:32.596864 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:32.596835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf98ba4-62e7-4cce-beea-2fd4a0b87e4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp\" (UID: \"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"
Jul 18 01:31:32.742580 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:32.742532    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"
Jul 18 01:31:32.891046 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:32.891016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"]
Jul 18 01:31:32.892915 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:31:32.892887    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf98ba4_62e7_4cce_beea_2fd4a0b87e4f.slice/crio-bb562a7aadb8b830907493d8e889c544365275676c810ae5b744f76e75136b06 WatchSource:0}: Error finding container bb562a7aadb8b830907493d8e889c544365275676c810ae5b744f76e75136b06: Status 404 returned error can't find the container with id bb562a7aadb8b830907493d8e889c544365275676c810ae5b744f76e75136b06
Jul 18 01:31:33.419673 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:33.419567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" event={"ID":"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f","Type":"ContainerStarted","Data":"ba1f15fdd9be758b6807ff40463df1b6f306337c26f29f559382549942ca6784"}
Jul 18 01:31:33.419673 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:33.419638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" event={"ID":"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f","Type":"ContainerStarted","Data":"bb562a7aadb8b830907493d8e889c544365275676c810ae5b744f76e75136b06"}
Jul 18 01:31:33.435653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:33.435590    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" podStartSLOduration=1.435569165 podStartE2EDuration="1.435569165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:31:33.433847859 +0000 UTC m=+3063.753825254" watchObservedRunningTime="2026-07-18 01:31:33.435569165 +0000 UTC m=+3063.755546598"
Jul 18 01:31:34.170118 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:34.170050    2563 scope.go:117] "RemoveContainer" containerID="c7f10cbfde57bee39c36cb65a4a40af917eb2839dd2518d635b4b5094ca85a4f"
Jul 18 01:31:34.427201 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:34.427112    2563 generic.go:356] "Generic (PLEG): container finished" podID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerID="ba1f15fdd9be758b6807ff40463df1b6f306337c26f29f559382549942ca6784" exitCode=6
Jul 18 01:31:34.427355 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:34.427198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" event={"ID":"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f","Type":"ContainerDied","Data":"ba1f15fdd9be758b6807ff40463df1b6f306337c26f29f559382549942ca6784"}
Jul 18 01:31:34.427355 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:34.427350    2563 scope.go:117] "RemoveContainer" containerID="ba1f15fdd9be758b6807ff40463df1b6f306337c26f29f559382549942ca6784"
Jul 18 01:31:35.435049 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:35.434959    2563 generic.go:356] "Generic (PLEG): container finished" podID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerID="7bcfece133070f532bab89fc7ff92746db5f51e5e6c1d9186bd2a439b5c7e69b" exitCode=6
Jul 18 01:31:35.435543 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:35.435048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" event={"ID":"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f","Type":"ContainerDied","Data":"7bcfece133070f532bab89fc7ff92746db5f51e5e6c1d9186bd2a439b5c7e69b"}
Jul 18 01:31:35.435543 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:35.435125    2563 scope.go:117] "RemoveContainer" containerID="ba1f15fdd9be758b6807ff40463df1b6f306337c26f29f559382549942ca6784"
Jul 18 01:31:35.435543 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:35.435343    2563 scope.go:117] "RemoveContainer" containerID="7bcfece133070f532bab89fc7ff92746db5f51e5e6c1d9186bd2a439b5c7e69b"
Jul 18 01:31:35.435683 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:35.435589    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp_odh-ai-gateway-infra(caf98ba4-62e7-4cce-beea-2fd4a0b87e4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f"
Jul 18 01:31:35.437114 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:35.437093    2563 generic.go:356] "Generic (PLEG): container finished" podID="9912333d-c04b-4eff-9420-53f29985813c" containerID="5f7e1930376f876b144fbe6921c67c958d7cb6a82fd7cdcdf0db2cae7e9a5633" exitCode=6
Jul 18 01:31:35.437224 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:35.437166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" event={"ID":"9912333d-c04b-4eff-9420-53f29985813c","Type":"ContainerDied","Data":"5f7e1930376f876b144fbe6921c67c958d7cb6a82fd7cdcdf0db2cae7e9a5633"}
Jul 18 01:31:35.437451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:35.437438    2563 scope.go:117] "RemoveContainer" containerID="5f7e1930376f876b144fbe6921c67c958d7cb6a82fd7cdcdf0db2cae7e9a5633"
Jul 18 01:31:35.437640 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:35.437623    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-6h6l6_odh-ai-gateway-infra(9912333d-c04b-4eff-9420-53f29985813c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" podUID="9912333d-c04b-4eff-9420-53f29985813c"
Jul 18 01:31:35.447244 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:35.447224    2563 scope.go:117] "RemoveContainer" containerID="c7f10cbfde57bee39c36cb65a4a40af917eb2839dd2518d635b4b5094ca85a4f"
Jul 18 01:31:36.456315 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:36.456287    2563 scope.go:117] "RemoveContainer" containerID="7bcfece133070f532bab89fc7ff92746db5f51e5e6c1d9186bd2a439b5c7e69b"
Jul 18 01:31:36.456721 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:36.456465    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp_odh-ai-gateway-infra(caf98ba4-62e7-4cce-beea-2fd4a0b87e4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f"
Jul 18 01:31:36.477367 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:36.477331    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"]
Jul 18 01:31:36.618244 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:36.618213    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"
Jul 18 01:31:36.626914 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:36.626891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912333d-c04b-4eff-9420-53f29985813c-maas-api-service-ca\") pod \"9912333d-c04b-4eff-9420-53f29985813c\" (UID: \"9912333d-c04b-4eff-9420-53f29985813c\") "
Jul 18 01:31:36.627247 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:36.627223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9912333d-c04b-4eff-9420-53f29985813c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9912333d-c04b-4eff-9420-53f29985813c" (UID: "9912333d-c04b-4eff-9420-53f29985813c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:31:36.727498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:36.727404    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912333d-c04b-4eff-9420-53f29985813c-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:31:37.355955 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.355905    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"]
Jul 18 01:31:37.356421 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.356402    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:31:37.356517 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.356423    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:31:37.356517 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.356436    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:31:37.356517 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.356444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:31:37.356517 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.356458    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:31:37.356517 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.356467    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:31:37.356782 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.356616    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:31:37.356782 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.356631    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:31:37.359922 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.359899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"
Jul 18 01:31:37.366480 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.366448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"]
Jul 18 01:31:37.431696 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.431656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf1d6c3-fd18-42ec-976c-4231da7097b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2\" (UID: \"abf1d6c3-fd18-42ec-976c-4231da7097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"
Jul 18 01:31:37.463555 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.463526    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"
Jul 18 01:31:37.463951 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.463525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6" event={"ID":"9912333d-c04b-4eff-9420-53f29985813c","Type":"ContainerDied","Data":"3398c90aa6b5daa617461bc725d2f62944bcfaef14b100216caf64a86c22598e"}
Jul 18 01:31:37.463951 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.463656    2563 scope.go:117] "RemoveContainer" containerID="5f7e1930376f876b144fbe6921c67c958d7cb6a82fd7cdcdf0db2cae7e9a5633"
Jul 18 01:31:37.487188 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.487153    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"]
Jul 18 01:31:37.492839 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.492806    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-6h6l6"]
Jul 18 01:31:37.532113 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.532073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf1d6c3-fd18-42ec-976c-4231da7097b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2\" (UID: \"abf1d6c3-fd18-42ec-976c-4231da7097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"
Jul 18 01:31:37.532761 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.532741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf1d6c3-fd18-42ec-976c-4231da7097b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2\" (UID: \"abf1d6c3-fd18-42ec-976c-4231da7097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"
Jul 18 01:31:37.673203 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.673112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"
Jul 18 01:31:37.818344 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:37.818315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"]
Jul 18 01:31:37.819848 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:31:37.819807    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf1d6c3_fd18_42ec_976c_4231da7097b6.slice/crio-5bdd8c448dff3159650e494fdaa2eabc77532d8031f453003061b3fc3030c4a6 WatchSource:0}: Error finding container 5bdd8c448dff3159650e494fdaa2eabc77532d8031f453003061b3fc3030c4a6: Status 404 returned error can't find the container with id 5bdd8c448dff3159650e494fdaa2eabc77532d8031f453003061b3fc3030c4a6
Jul 18 01:31:38.170275 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:38.170243    2563 scope.go:117] "RemoveContainer" containerID="75f56e1dc20bfd2403ff7a0e5ee608ebbd6f9e39772a43ebd8fa5a22ed666464"
Jul 18 01:31:38.177810 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:38.177415    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9912333d-c04b-4eff-9420-53f29985813c" path="/var/lib/kubelet/pods/9912333d-c04b-4eff-9420-53f29985813c/volumes"
Jul 18 01:31:38.471098 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:38.471045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" event={"ID":"abf1d6c3-fd18-42ec-976c-4231da7097b6","Type":"ContainerStarted","Data":"0fcfe346ee7c107b15cc5c7905d7973ef77426e499779119863eb3bd6548f7e6"}
Jul 18 01:31:38.471531 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:38.471125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" event={"ID":"abf1d6c3-fd18-42ec-976c-4231da7097b6","Type":"ContainerStarted","Data":"5bdd8c448dff3159650e494fdaa2eabc77532d8031f453003061b3fc3030c4a6"}
Jul 18 01:31:38.485342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:38.485284    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" podStartSLOduration=1.485263503 podStartE2EDuration="1.485263503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:31:38.484880982 +0000 UTC m=+3068.804858364" watchObservedRunningTime="2026-07-18 01:31:38.485263503 +0000 UTC m=+3068.805240888"
Jul 18 01:31:39.479918 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:39.479881    2563 generic.go:356] "Generic (PLEG): container finished" podID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerID="0fcfe346ee7c107b15cc5c7905d7973ef77426e499779119863eb3bd6548f7e6" exitCode=6
Jul 18 01:31:39.480492 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:39.479966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" event={"ID":"abf1d6c3-fd18-42ec-976c-4231da7097b6","Type":"ContainerDied","Data":"0fcfe346ee7c107b15cc5c7905d7973ef77426e499779119863eb3bd6548f7e6"}
Jul 18 01:31:39.480492 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:39.480184    2563 scope.go:117] "RemoveContainer" containerID="0fcfe346ee7c107b15cc5c7905d7973ef77426e499779119863eb3bd6548f7e6"
Jul 18 01:31:39.482263 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:39.482239    2563 generic.go:356] "Generic (PLEG): container finished" podID="ddba1201-18be-4333-b963-3a6f9abf179a" containerID="6aa568e4a461e3bf7fbc7b10713f00331feea744298e3c4b9af82a2c149ae539" exitCode=6
Jul 18 01:31:39.482377 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:39.482306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" event={"ID":"ddba1201-18be-4333-b963-3a6f9abf179a","Type":"ContainerDied","Data":"6aa568e4a461e3bf7fbc7b10713f00331feea744298e3c4b9af82a2c149ae539"}
Jul 18 01:31:39.482377 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:39.482370    2563 scope.go:117] "RemoveContainer" containerID="75f56e1dc20bfd2403ff7a0e5ee608ebbd6f9e39772a43ebd8fa5a22ed666464"
Jul 18 01:31:39.482778 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:39.482658    2563 scope.go:117] "RemoveContainer" containerID="6aa568e4a461e3bf7fbc7b10713f00331feea744298e3c4b9af82a2c149ae539"
Jul 18 01:31:39.482908 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:39.482882    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl_odh-ai-gateway-infra(ddba1201-18be-4333-b963-3a6f9abf179a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" podUID="ddba1201-18be-4333-b963-3a6f9abf179a"
Jul 18 01:31:40.493164 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.493130    2563 generic.go:356] "Generic (PLEG): container finished" podID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerID="9f0bdca165a7f40dfd89bb6f14720ac1ea8aa2e1d569fd372f626a97c02138ac" exitCode=6
Jul 18 01:31:40.493653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.493206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" event={"ID":"abf1d6c3-fd18-42ec-976c-4231da7097b6","Type":"ContainerDied","Data":"9f0bdca165a7f40dfd89bb6f14720ac1ea8aa2e1d569fd372f626a97c02138ac"}
Jul 18 01:31:40.493653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.493254    2563 scope.go:117] "RemoveContainer" containerID="0fcfe346ee7c107b15cc5c7905d7973ef77426e499779119863eb3bd6548f7e6"
Jul 18 01:31:40.493653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.493491    2563 scope.go:117] "RemoveContainer" containerID="9f0bdca165a7f40dfd89bb6f14720ac1ea8aa2e1d569fd372f626a97c02138ac"
Jul 18 01:31:40.493816 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:40.493765    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2_odh-ai-gateway-infra(abf1d6c3-fd18-42ec-976c-4231da7097b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6"
Jul 18 01:31:40.519015 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.518977    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"]
Jul 18 01:31:40.663032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.663004    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"
Jul 18 01:31:40.759413 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.759321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddba1201-18be-4333-b963-3a6f9abf179a-maas-api-service-ca\") pod \"ddba1201-18be-4333-b963-3a6f9abf179a\" (UID: \"ddba1201-18be-4333-b963-3a6f9abf179a\") "
Jul 18 01:31:40.759707 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.759680    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddba1201-18be-4333-b963-3a6f9abf179a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddba1201-18be-4333-b963-3a6f9abf179a" (UID: "ddba1201-18be-4333-b963-3a6f9abf179a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:31:40.832028 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.831992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:31:40.860025 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:40.859990    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddba1201-18be-4333-b963-3a6f9abf179a-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:31:41.501798 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:41.501771    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"
Jul 18 01:31:41.502311 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:41.501820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl" event={"ID":"ddba1201-18be-4333-b963-3a6f9abf179a","Type":"ContainerDied","Data":"106c2ced4f3d17bdb9ee9caf1663f9c948ade1b5dfd1706bb4680e45a8b26d76"}
Jul 18 01:31:41.502311 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:41.501881    2563 scope.go:117] "RemoveContainer" containerID="6aa568e4a461e3bf7fbc7b10713f00331feea744298e3c4b9af82a2c149ae539"
Jul 18 01:31:41.503845 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:41.503829    2563 scope.go:117] "RemoveContainer" containerID="9f0bdca165a7f40dfd89bb6f14720ac1ea8aa2e1d569fd372f626a97c02138ac"
Jul 18 01:31:41.504023 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:41.504005    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2_odh-ai-gateway-infra(abf1d6c3-fd18-42ec-976c-4231da7097b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6"
Jul 18 01:31:41.537529 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:41.537491    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"]
Jul 18 01:31:41.548417 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:41.545760    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-7ccfl"]
Jul 18 01:31:42.176188 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:42.176144    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" path="/var/lib/kubelet/pods/ddba1201-18be-4333-b963-3a6f9abf179a/volumes"
Jul 18 01:31:50.172396 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:50.172363    2563 scope.go:117] "RemoveContainer" containerID="7bcfece133070f532bab89fc7ff92746db5f51e5e6c1d9186bd2a439b5c7e69b"
Jul 18 01:31:50.561983 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:50.561945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" event={"ID":"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f","Type":"ContainerStarted","Data":"6a3efd9e634a45df8be46a0285ed1f17d5e77dc260f7ec76e1b2962b6b5f5e38"}
Jul 18 01:31:51.570998 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:51.570964    2563 generic.go:356] "Generic (PLEG): container finished" podID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerID="6a3efd9e634a45df8be46a0285ed1f17d5e77dc260f7ec76e1b2962b6b5f5e38" exitCode=6
Jul 18 01:31:51.571434 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:51.571045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" event={"ID":"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f","Type":"ContainerDied","Data":"6a3efd9e634a45df8be46a0285ed1f17d5e77dc260f7ec76e1b2962b6b5f5e38"}
Jul 18 01:31:51.571434 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:51.571145    2563 scope.go:117] "RemoveContainer" containerID="7bcfece133070f532bab89fc7ff92746db5f51e5e6c1d9186bd2a439b5c7e69b"
Jul 18 01:31:51.571575 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:51.571478    2563 scope.go:117] "RemoveContainer" containerID="6a3efd9e634a45df8be46a0285ed1f17d5e77dc260f7ec76e1b2962b6b5f5e38"
Jul 18 01:31:51.571718 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:51.571698    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp_odh-ai-gateway-infra(caf98ba4-62e7-4cce-beea-2fd4a0b87e4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f"
Jul 18 01:31:51.586221 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:51.586195    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"]
Jul 18 01:31:52.718275 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:52.718246    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"
Jul 18 01:31:52.766949 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:52.766916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf98ba4-62e7-4cce-beea-2fd4a0b87e4f-maas-api-service-ca\") pod \"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f\" (UID: \"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f\") "
Jul 18 01:31:52.767327 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:52.767303    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf98ba4-62e7-4cce-beea-2fd4a0b87e4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" (UID: "caf98ba4-62e7-4cce-beea-2fd4a0b87e4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:31:52.867803 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:52.867707    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf98ba4-62e7-4cce-beea-2fd4a0b87e4f-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:31:53.586616 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:53.586578    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"
Jul 18 01:31:53.586616 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:53.586594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp" event={"ID":"caf98ba4-62e7-4cce-beea-2fd4a0b87e4f","Type":"ContainerDied","Data":"bb562a7aadb8b830907493d8e889c544365275676c810ae5b744f76e75136b06"}
Jul 18 01:31:53.586896 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:53.586662    2563 scope.go:117] "RemoveContainer" containerID="6a3efd9e634a45df8be46a0285ed1f17d5e77dc260f7ec76e1b2962b6b5f5e38"
Jul 18 01:31:53.610236 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:53.610205    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"]
Jul 18 01:31:53.616049 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:53.616018    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-qbhkp"]
Jul 18 01:31:54.178048 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:54.177999    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" path="/var/lib/kubelet/pods/caf98ba4-62e7-4cce-beea-2fd4a0b87e4f/volumes"
Jul 18 01:31:55.435362 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:55.435319    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:31:56.170349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:56.170313    2563 scope.go:117] "RemoveContainer" containerID="9f0bdca165a7f40dfd89bb6f14720ac1ea8aa2e1d569fd372f626a97c02138ac"
Jul 18 01:31:56.608843 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:56.608807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" event={"ID":"abf1d6c3-fd18-42ec-976c-4231da7097b6","Type":"ContainerStarted","Data":"9559d622ce6bf80bb24d4725303dfc57a4b5e9beeef901860669dfcf101b452e"}
Jul 18 01:31:57.617282 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:57.617243    2563 generic.go:356] "Generic (PLEG): container finished" podID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerID="9559d622ce6bf80bb24d4725303dfc57a4b5e9beeef901860669dfcf101b452e" exitCode=6
Jul 18 01:31:57.617740 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:57.617295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" event={"ID":"abf1d6c3-fd18-42ec-976c-4231da7097b6","Type":"ContainerDied","Data":"9559d622ce6bf80bb24d4725303dfc57a4b5e9beeef901860669dfcf101b452e"}
Jul 18 01:31:57.617740 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:57.617348    2563 scope.go:117] "RemoveContainer" containerID="9f0bdca165a7f40dfd89bb6f14720ac1ea8aa2e1d569fd372f626a97c02138ac"
Jul 18 01:31:57.617740 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:57.617623    2563 scope.go:117] "RemoveContainer" containerID="9559d622ce6bf80bb24d4725303dfc57a4b5e9beeef901860669dfcf101b452e"
Jul 18 01:31:57.617887 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:31:57.617864    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2_odh-ai-gateway-infra(abf1d6c3-fd18-42ec-976c-4231da7097b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6"
Jul 18 01:31:57.633681 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:57.633652    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"]
Jul 18 01:31:58.766209 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:58.766179    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"
Jul 18 01:31:58.829185 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:58.829149    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf1d6c3-fd18-42ec-976c-4231da7097b6-maas-api-service-ca\") pod \"abf1d6c3-fd18-42ec-976c-4231da7097b6\" (UID: \"abf1d6c3-fd18-42ec-976c-4231da7097b6\") "
Jul 18 01:31:58.829520 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:58.829492    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf1d6c3-fd18-42ec-976c-4231da7097b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abf1d6c3-fd18-42ec-976c-4231da7097b6" (UID: "abf1d6c3-fd18-42ec-976c-4231da7097b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:31:58.930216 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:58.930124    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf1d6c3-fd18-42ec-976c-4231da7097b6-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:31:59.639845 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:59.639809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"
Jul 18 01:31:59.639845 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:59.639831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2" event={"ID":"abf1d6c3-fd18-42ec-976c-4231da7097b6","Type":"ContainerDied","Data":"5bdd8c448dff3159650e494fdaa2eabc77532d8031f453003061b3fc3030c4a6"}
Jul 18 01:31:59.640044 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:59.639925    2563 scope.go:117] "RemoveContainer" containerID="9559d622ce6bf80bb24d4725303dfc57a4b5e9beeef901860669dfcf101b452e"
Jul 18 01:31:59.661654 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:59.661616    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"]
Jul 18 01:31:59.665869 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:31:59.665836    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-6tjw2"]
Jul 18 01:32:00.175711 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:00.175677    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" path="/var/lib/kubelet/pods/abf1d6c3-fd18-42ec-976c-4231da7097b6/volumes"
Jul 18 01:32:13.436953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:13.436913    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:32:15.385463 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385422    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"]
Jul 18 01:32:15.385854 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385812    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:15.385854 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385823    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:15.385854 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385835    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:15.385854 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385840    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:15.385854 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385853    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385859    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385867    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385873    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385880    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385885    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385894    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385899    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385907    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385967    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385973    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385980    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385987    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf1d6c3-fd18-42ec-976c-4231da7097b6" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385993    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.385998    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9912333d-c04b-4eff-9420-53f29985813c" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.386003    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:15.386019 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.386008    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:15.388754 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.388736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"
Jul 18 01:32:15.391173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.391152    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:32:15.398687 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.398657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"]
Jul 18 01:32:15.475502 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.475460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd0ed86-ba54-49dd-93cb-14d26fbd4931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-kjhd2\" (UID: \"6fd0ed86-ba54-49dd-93cb-14d26fbd4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"
Jul 18 01:32:15.576884 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.576838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd0ed86-ba54-49dd-93cb-14d26fbd4931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-kjhd2\" (UID: \"6fd0ed86-ba54-49dd-93cb-14d26fbd4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"
Jul 18 01:32:15.577560 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.577533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd0ed86-ba54-49dd-93cb-14d26fbd4931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-kjhd2\" (UID: \"6fd0ed86-ba54-49dd-93cb-14d26fbd4931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"
Jul 18 01:32:15.701417 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.701387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"
Jul 18 01:32:15.842106 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:15.842077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"]
Jul 18 01:32:15.844181 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:32:15.844142    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd0ed86_ba54_49dd_93cb_14d26fbd4931.slice/crio-ecb9653ba5b957abf0f7cca4073c6e5570ba9dc10fd24282140b9ab59171169c WatchSource:0}: Error finding container ecb9653ba5b957abf0f7cca4073c6e5570ba9dc10fd24282140b9ab59171169c: Status 404 returned error can't find the container with id ecb9653ba5b957abf0f7cca4073c6e5570ba9dc10fd24282140b9ab59171169c
Jul 18 01:32:16.742804 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:16.742711    2563 generic.go:356] "Generic (PLEG): container finished" podID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerID="0a7fc477063669a81c631c88a1cb0a5c8082c719aaa7e848305b45eabd358803" exitCode=6
Jul 18 01:32:16.742804 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:16.742769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" event={"ID":"6fd0ed86-ba54-49dd-93cb-14d26fbd4931","Type":"ContainerDied","Data":"0a7fc477063669a81c631c88a1cb0a5c8082c719aaa7e848305b45eabd358803"}
Jul 18 01:32:16.743246 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:16.742817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" event={"ID":"6fd0ed86-ba54-49dd-93cb-14d26fbd4931","Type":"ContainerStarted","Data":"ecb9653ba5b957abf0f7cca4073c6e5570ba9dc10fd24282140b9ab59171169c"}
Jul 18 01:32:16.743246 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:16.743038    2563 scope.go:117] "RemoveContainer" containerID="0a7fc477063669a81c631c88a1cb0a5c8082c719aaa7e848305b45eabd358803"
Jul 18 01:32:17.750418 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:17.750328    2563 generic.go:356] "Generic (PLEG): container finished" podID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerID="cf3535bcc42981a34cbe8b977596aa18ec38800bcc6b429b65467643adaf8c2d" exitCode=6
Jul 18 01:32:17.750799 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:17.750415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" event={"ID":"6fd0ed86-ba54-49dd-93cb-14d26fbd4931","Type":"ContainerDied","Data":"cf3535bcc42981a34cbe8b977596aa18ec38800bcc6b429b65467643adaf8c2d"}
Jul 18 01:32:17.750799 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:17.750473    2563 scope.go:117] "RemoveContainer" containerID="0a7fc477063669a81c631c88a1cb0a5c8082c719aaa7e848305b45eabd358803"
Jul 18 01:32:17.750799 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:17.750603    2563 scope.go:117] "RemoveContainer" containerID="cf3535bcc42981a34cbe8b977596aa18ec38800bcc6b429b65467643adaf8c2d"
Jul 18 01:32:17.750932 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:17.750829    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-kjhd2_odh-ai-gateway-infra(6fd0ed86-ba54-49dd-93cb-14d26fbd4931)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931"
Jul 18 01:32:18.759222 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:18.759188    2563 scope.go:117] "RemoveContainer" containerID="cf3535bcc42981a34cbe8b977596aa18ec38800bcc6b429b65467643adaf8c2d"
Jul 18 01:32:18.759707 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:18.759462    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-kjhd2_odh-ai-gateway-infra(6fd0ed86-ba54-49dd-93cb-14d26fbd4931)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931"
Jul 18 01:32:19.328538 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.328500    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"]
Jul 18 01:32:19.328925 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.328912    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:19.328971 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.328927    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:19.328971 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.328936    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:19.328971 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.328943    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:19.329094 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.329005    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf98ba4-62e7-4cce-beea-2fd4a0b87e4f" containerName="revoke-keys"
Jul 18 01:32:19.329094 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.329014    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddba1201-18be-4333-b963-3a6f9abf179a" containerName="revoke-keys"
Jul 18 01:32:19.333291 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.333271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"
Jul 18 01:32:19.340264 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.340233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"]
Jul 18 01:32:19.421297 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.421253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20345cc3-5abc-453b-8e70-6c475c88d8e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9\" (UID: \"20345cc3-5abc-453b-8e70-6c475c88d8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"
Jul 18 01:32:19.522497 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.522449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20345cc3-5abc-453b-8e70-6c475c88d8e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9\" (UID: \"20345cc3-5abc-453b-8e70-6c475c88d8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"
Jul 18 01:32:19.523135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.523115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20345cc3-5abc-453b-8e70-6c475c88d8e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9\" (UID: \"20345cc3-5abc-453b-8e70-6c475c88d8e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"
Jul 18 01:32:19.645051 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.644969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"
Jul 18 01:32:19.789363 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:19.789332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"]
Jul 18 01:32:19.791866 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:32:19.791838    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20345cc3_5abc_453b_8e70_6c475c88d8e2.slice/crio-fefc0723c907f02089d789f0d63d77e093725ad878d425885e42335265964553 WatchSource:0}: Error finding container fefc0723c907f02089d789f0d63d77e093725ad878d425885e42335265964553: Status 404 returned error can't find the container with id fefc0723c907f02089d789f0d63d77e093725ad878d425885e42335265964553
Jul 18 01:32:20.777110 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:20.777041    2563 generic.go:356] "Generic (PLEG): container finished" podID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerID="507a5ff1387edbd3bf4e669314e3e4f0b38242baed8ef43ea217a4afa72bfcf1" exitCode=6
Jul 18 01:32:20.777317 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:20.777126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" event={"ID":"20345cc3-5abc-453b-8e70-6c475c88d8e2","Type":"ContainerDied","Data":"507a5ff1387edbd3bf4e669314e3e4f0b38242baed8ef43ea217a4afa72bfcf1"}
Jul 18 01:32:20.777317 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:20.777182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" event={"ID":"20345cc3-5abc-453b-8e70-6c475c88d8e2","Type":"ContainerStarted","Data":"fefc0723c907f02089d789f0d63d77e093725ad878d425885e42335265964553"}
Jul 18 01:32:20.777317 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:20.777296    2563 scope.go:117] "RemoveContainer" containerID="507a5ff1387edbd3bf4e669314e3e4f0b38242baed8ef43ea217a4afa72bfcf1"
Jul 18 01:32:21.787281 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:21.787174    2563 generic.go:356] "Generic (PLEG): container finished" podID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerID="d75dd157130a6c0162f670342db317bde6803154953ca7e86d3d77d8103d879a" exitCode=6
Jul 18 01:32:21.787281 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:21.787260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" event={"ID":"20345cc3-5abc-453b-8e70-6c475c88d8e2","Type":"ContainerDied","Data":"d75dd157130a6c0162f670342db317bde6803154953ca7e86d3d77d8103d879a"}
Jul 18 01:32:21.787748 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:21.787321    2563 scope.go:117] "RemoveContainer" containerID="507a5ff1387edbd3bf4e669314e3e4f0b38242baed8ef43ea217a4afa72bfcf1"
Jul 18 01:32:21.787748 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:21.787463    2563 scope.go:117] "RemoveContainer" containerID="d75dd157130a6c0162f670342db317bde6803154953ca7e86d3d77d8103d879a"
Jul 18 01:32:21.787748 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:21.787694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9_odh-ai-gateway-infra(20345cc3-5abc-453b-8e70-6c475c88d8e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2"
Jul 18 01:32:22.795779 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:22.795748    2563 scope.go:117] "RemoveContainer" containerID="d75dd157130a6c0162f670342db317bde6803154953ca7e86d3d77d8103d879a"
Jul 18 01:32:22.796195 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:22.795949    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9_odh-ai-gateway-infra(20345cc3-5abc-453b-8e70-6c475c88d8e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2"
Jul 18 01:32:31.545528 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:31.545490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:32:32.465529 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:32.465489    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"]
Jul 18 01:32:32.470781 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:32.470754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"
Jul 18 01:32:32.475634 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:32.475590    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"]
Jul 18 01:32:32.537277 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:32.537238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70ffe71-8bb4-4554-a038-6501f4fa22a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm\" (UID: \"a70ffe71-8bb4-4554-a038-6501f4fa22a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"
Jul 18 01:32:32.638611 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:32.638562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70ffe71-8bb4-4554-a038-6501f4fa22a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm\" (UID: \"a70ffe71-8bb4-4554-a038-6501f4fa22a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"
Jul 18 01:32:32.639251 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:32.639227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70ffe71-8bb4-4554-a038-6501f4fa22a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm\" (UID: \"a70ffe71-8bb4-4554-a038-6501f4fa22a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"
Jul 18 01:32:32.784370 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:32.784261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"
Jul 18 01:32:32.928185 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:32.928156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"]
Jul 18 01:32:32.929782 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:32:32.929747    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70ffe71_8bb4_4554_a038_6501f4fa22a0.slice/crio-4391f63a4cfcb069b7fa0b0df023f85ac3ad46f5c9c5919019a8386efa9672ae WatchSource:0}: Error finding container 4391f63a4cfcb069b7fa0b0df023f85ac3ad46f5c9c5919019a8386efa9672ae: Status 404 returned error can't find the container with id 4391f63a4cfcb069b7fa0b0df023f85ac3ad46f5c9c5919019a8386efa9672ae
Jul 18 01:32:33.863413 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:33.863314    2563 generic.go:356] "Generic (PLEG): container finished" podID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerID="989eadd6c17bbadbb26dcd84a6a63e17e89e33588e4482ef23b46a0b07ecfc42" exitCode=6
Jul 18 01:32:33.863808 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:33.863398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm" event={"ID":"a70ffe71-8bb4-4554-a038-6501f4fa22a0","Type":"ContainerDied","Data":"989eadd6c17bbadbb26dcd84a6a63e17e89e33588e4482ef23b46a0b07ecfc42"}
Jul 18 01:32:33.863808 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:33.863467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm" event={"ID":"a70ffe71-8bb4-4554-a038-6501f4fa22a0","Type":"ContainerStarted","Data":"4391f63a4cfcb069b7fa0b0df023f85ac3ad46f5c9c5919019a8386efa9672ae"}
Jul 18 01:32:33.863808 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:33.863594    2563 scope.go:117] "RemoveContainer" containerID="989eadd6c17bbadbb26dcd84a6a63e17e89e33588e4482ef23b46a0b07ecfc42"
Jul 18 01:32:34.170293 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.170196    2563 scope.go:117] "RemoveContainer" containerID="d75dd157130a6c0162f670342db317bde6803154953ca7e86d3d77d8103d879a"
Jul 18 01:32:34.170470 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.170326    2563 scope.go:117] "RemoveContainer" containerID="cf3535bcc42981a34cbe8b977596aa18ec38800bcc6b429b65467643adaf8c2d"
Jul 18 01:32:34.870841 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.870740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" event={"ID":"20345cc3-5abc-453b-8e70-6c475c88d8e2","Type":"ContainerStarted","Data":"3aa6212fad49409fa4a4786869e02db2652d661f6274068b159c405845add6aa"}
Jul 18 01:32:34.872719 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.872694    2563 generic.go:356] "Generic (PLEG): container finished" podID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerID="e6c4d6da5abfe46a490847ba30170fa0162d1070545fe5abc77fa5b2767f2734" exitCode=6
Jul 18 01:32:34.872880 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.872770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm" event={"ID":"a70ffe71-8bb4-4554-a038-6501f4fa22a0","Type":"ContainerDied","Data":"e6c4d6da5abfe46a490847ba30170fa0162d1070545fe5abc77fa5b2767f2734"}
Jul 18 01:32:34.872880 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.872828    2563 scope.go:117] "RemoveContainer" containerID="989eadd6c17bbadbb26dcd84a6a63e17e89e33588e4482ef23b46a0b07ecfc42"
Jul 18 01:32:34.873017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.872997    2563 scope.go:117] "RemoveContainer" containerID="e6c4d6da5abfe46a490847ba30170fa0162d1070545fe5abc77fa5b2767f2734"
Jul 18 01:32:34.873301 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:34.873276    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm_odh-ai-gateway-infra(a70ffe71-8bb4-4554-a038-6501f4fa22a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0"
Jul 18 01:32:34.874664 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.874643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" event={"ID":"6fd0ed86-ba54-49dd-93cb-14d26fbd4931","Type":"ContainerStarted","Data":"4219be740938a0e7ad08bf4d7cf6802b73bec7336b8685ceedc3e15eb7c3a879"}
Jul 18 01:32:34.887122 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.887049    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" podStartSLOduration=15.887031408 podStartE2EDuration="15.887031408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:32:34.884790029 +0000 UTC m=+3125.204767413" watchObservedRunningTime="2026-07-18 01:32:34.887031408 +0000 UTC m=+3125.207008791"
Jul 18 01:32:34.901251 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:34.901199    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" podStartSLOduration=19.901182475 podStartE2EDuration="19.901182475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:32:34.89852632 +0000 UTC m=+3125.218503704" watchObservedRunningTime="2026-07-18 01:32:34.901182475 +0000 UTC m=+3125.221159877"
Jul 18 01:32:35.113537 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:35.113504    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd0ed86_ba54_49dd_93cb_14d26fbd4931.slice/crio-4219be740938a0e7ad08bf4d7cf6802b73bec7336b8685ceedc3e15eb7c3a879.scope\": RecentStats: unable to find data in memory cache]"
Jul 18 01:32:35.883154 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.883115    2563 generic.go:356] "Generic (PLEG): container finished" podID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerID="4219be740938a0e7ad08bf4d7cf6802b73bec7336b8685ceedc3e15eb7c3a879" exitCode=6
Jul 18 01:32:35.883604 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.883196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" event={"ID":"6fd0ed86-ba54-49dd-93cb-14d26fbd4931","Type":"ContainerDied","Data":"4219be740938a0e7ad08bf4d7cf6802b73bec7336b8685ceedc3e15eb7c3a879"}
Jul 18 01:32:35.883604 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.883309    2563 scope.go:117] "RemoveContainer" containerID="cf3535bcc42981a34cbe8b977596aa18ec38800bcc6b429b65467643adaf8c2d"
Jul 18 01:32:35.883760 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.883616    2563 scope.go:117] "RemoveContainer" containerID="4219be740938a0e7ad08bf4d7cf6802b73bec7336b8685ceedc3e15eb7c3a879"
Jul 18 01:32:35.883896 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:35.883866    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-kjhd2_odh-ai-gateway-infra(6fd0ed86-ba54-49dd-93cb-14d26fbd4931)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931"
Jul 18 01:32:35.885338 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.885314    2563 generic.go:356] "Generic (PLEG): container finished" podID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerID="3aa6212fad49409fa4a4786869e02db2652d661f6274068b159c405845add6aa" exitCode=6
Jul 18 01:32:35.885432 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.885390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" event={"ID":"20345cc3-5abc-453b-8e70-6c475c88d8e2","Type":"ContainerDied","Data":"3aa6212fad49409fa4a4786869e02db2652d661f6274068b159c405845add6aa"}
Jul 18 01:32:35.885708 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.885692    2563 scope.go:117] "RemoveContainer" containerID="3aa6212fad49409fa4a4786869e02db2652d661f6274068b159c405845add6aa"
Jul 18 01:32:35.885923 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:35.885902    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9_odh-ai-gateway-infra(20345cc3-5abc-453b-8e70-6c475c88d8e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2"
Jul 18 01:32:35.887236 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.887217    2563 scope.go:117] "RemoveContainer" containerID="e6c4d6da5abfe46a490847ba30170fa0162d1070545fe5abc77fa5b2767f2734"
Jul 18 01:32:35.887449 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:35.887424    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm_odh-ai-gateway-infra(a70ffe71-8bb4-4554-a038-6501f4fa22a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0"
Jul 18 01:32:35.895657 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.895635    2563 scope.go:117] "RemoveContainer" containerID="d75dd157130a6c0162f670342db317bde6803154953ca7e86d3d77d8103d879a"
Jul 18 01:32:35.899194 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.899171    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"]
Jul 18 01:32:35.911594 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:35.911569    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"]
Jul 18 01:32:37.039184 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.039152    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"
Jul 18 01:32:37.076805 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.076779    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"
Jul 18 01:32:37.076939 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.076914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20345cc3-5abc-453b-8e70-6c475c88d8e2-maas-api-service-ca\") pod \"20345cc3-5abc-453b-8e70-6c475c88d8e2\" (UID: \"20345cc3-5abc-453b-8e70-6c475c88d8e2\") "
Jul 18 01:32:37.077344 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.077320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20345cc3-5abc-453b-8e70-6c475c88d8e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "20345cc3-5abc-453b-8e70-6c475c88d8e2" (UID: "20345cc3-5abc-453b-8e70-6c475c88d8e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:32:37.178471 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.178358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd0ed86-ba54-49dd-93cb-14d26fbd4931-maas-api-service-ca\") pod \"6fd0ed86-ba54-49dd-93cb-14d26fbd4931\" (UID: \"6fd0ed86-ba54-49dd-93cb-14d26fbd4931\") "
Jul 18 01:32:37.178684 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.178663    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20345cc3-5abc-453b-8e70-6c475c88d8e2-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:32:37.178756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.178719    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd0ed86-ba54-49dd-93cb-14d26fbd4931-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fd0ed86-ba54-49dd-93cb-14d26fbd4931" (UID: "6fd0ed86-ba54-49dd-93cb-14d26fbd4931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:32:37.280106 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.280040    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd0ed86-ba54-49dd-93cb-14d26fbd4931-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:32:37.403733 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.403688    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"]
Jul 18 01:32:37.404211 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404192    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:37.404211 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404212    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:37.404373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:37.404373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404236    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:37.404373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404249    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:37.404373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404257    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:37.404373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404273    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:37.404373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404281    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:37.404373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404292    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:37.404373 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404300    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:37.404763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404393    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:37.404763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404407    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:37.404763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404420    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:37.404763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404429    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:37.404763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.404437    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:37.407822 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.407799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"
Jul 18 01:32:37.414043 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.414007    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"]
Jul 18 01:32:37.482101 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.482042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9051b65-4c29-4f2a-b8f8-d859a924f0d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk\" (UID: \"e9051b65-4c29-4f2a-b8f8-d859a924f0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"
Jul 18 01:32:37.583136 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.583082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9051b65-4c29-4f2a-b8f8-d859a924f0d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk\" (UID: \"e9051b65-4c29-4f2a-b8f8-d859a924f0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"
Jul 18 01:32:37.583699 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.583678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9051b65-4c29-4f2a-b8f8-d859a924f0d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk\" (UID: \"e9051b65-4c29-4f2a-b8f8-d859a924f0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"
Jul 18 01:32:37.720447 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.720405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"
Jul 18 01:32:37.888004 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.887974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"]
Jul 18 01:32:37.889270 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:32:37.889240    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9051b65_4c29_4f2a_b8f8_d859a924f0d4.slice/crio-2468675b3d4b8f1204c8d7461e9ffd5e37edc33584727d5787fc4110a6f601f5 WatchSource:0}: Error finding container 2468675b3d4b8f1204c8d7461e9ffd5e37edc33584727d5787fc4110a6f601f5: Status 404 returned error can't find the container with id 2468675b3d4b8f1204c8d7461e9ffd5e37edc33584727d5787fc4110a6f601f5
Jul 18 01:32:37.903624 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.903596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk" event={"ID":"e9051b65-4c29-4f2a-b8f8-d859a924f0d4","Type":"ContainerStarted","Data":"2468675b3d4b8f1204c8d7461e9ffd5e37edc33584727d5787fc4110a6f601f5"}
Jul 18 01:32:37.905025 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.905010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"
Jul 18 01:32:37.905129 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.905018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2" event={"ID":"6fd0ed86-ba54-49dd-93cb-14d26fbd4931","Type":"ContainerDied","Data":"ecb9653ba5b957abf0f7cca4073c6e5570ba9dc10fd24282140b9ab59171169c"}
Jul 18 01:32:37.905129 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.905098    2563 scope.go:117] "RemoveContainer" containerID="4219be740938a0e7ad08bf4d7cf6802b73bec7336b8685ceedc3e15eb7c3a879"
Jul 18 01:32:37.906536 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.906503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9" event={"ID":"20345cc3-5abc-453b-8e70-6c475c88d8e2","Type":"ContainerDied","Data":"fefc0723c907f02089d789f0d63d77e093725ad878d425885e42335265964553"}
Jul 18 01:32:37.906638 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.906543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"
Jul 18 01:32:37.915529 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.915511    2563 scope.go:117] "RemoveContainer" containerID="3aa6212fad49409fa4a4786869e02db2652d661f6274068b159c405845add6aa"
Jul 18 01:32:37.943614 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.943577    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"]
Jul 18 01:32:37.949265 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.949231    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-4s8n9"]
Jul 18 01:32:37.958756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.958722    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"]
Jul 18 01:32:37.967403 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:37.967374    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-kjhd2"]
Jul 18 01:32:38.176783 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:38.176745    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" path="/var/lib/kubelet/pods/20345cc3-5abc-453b-8e70-6c475c88d8e2/volumes"
Jul 18 01:32:38.177180 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:38.177157    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" path="/var/lib/kubelet/pods/6fd0ed86-ba54-49dd-93cb-14d26fbd4931/volumes"
Jul 18 01:32:38.917459 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:38.917356    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerID="86196bce7480caed96d8fb447ab3c2e749c054a8939d557356b8ce0c630ded5d" exitCode=6
Jul 18 01:32:38.917459 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:38.917446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk" event={"ID":"e9051b65-4c29-4f2a-b8f8-d859a924f0d4","Type":"ContainerDied","Data":"86196bce7480caed96d8fb447ab3c2e749c054a8939d557356b8ce0c630ded5d"}
Jul 18 01:32:38.917720 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:38.917703    2563 scope.go:117] "RemoveContainer" containerID="86196bce7480caed96d8fb447ab3c2e749c054a8939d557356b8ce0c630ded5d"
Jul 18 01:32:39.927528 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:39.927434    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerID="95b4cdd2c16cf93385609abf8cd022a4373332d3c33a3f1407ace41488956384" exitCode=6
Jul 18 01:32:39.927980 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:39.927525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk" event={"ID":"e9051b65-4c29-4f2a-b8f8-d859a924f0d4","Type":"ContainerDied","Data":"95b4cdd2c16cf93385609abf8cd022a4373332d3c33a3f1407ace41488956384"}
Jul 18 01:32:39.927980 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:39.927642    2563 scope.go:117] "RemoveContainer" containerID="86196bce7480caed96d8fb447ab3c2e749c054a8939d557356b8ce0c630ded5d"
Jul 18 01:32:39.927980 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:39.927855    2563 scope.go:117] "RemoveContainer" containerID="95b4cdd2c16cf93385609abf8cd022a4373332d3c33a3f1407ace41488956384"
Jul 18 01:32:39.928148 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:39.928127    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk_odh-ai-gateway-infra(e9051b65-4c29-4f2a-b8f8-d859a924f0d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4"
Jul 18 01:32:40.935657 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:40.935629    2563 scope.go:117] "RemoveContainer" containerID="95b4cdd2c16cf93385609abf8cd022a4373332d3c33a3f1407ace41488956384"
Jul 18 01:32:40.936049 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:40.935865    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk_odh-ai-gateway-infra(e9051b65-4c29-4f2a-b8f8-d859a924f0d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4"
Jul 18 01:32:48.832091 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:48.832033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:32:50.174337 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:50.174309    2563 scope.go:117] "RemoveContainer" containerID="e6c4d6da5abfe46a490847ba30170fa0162d1070545fe5abc77fa5b2767f2734"
Jul 18 01:32:50.999619 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:50.999581    2563 generic.go:356] "Generic (PLEG): container finished" podID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerID="b98aaa7489932eba488eee72431637c924c7073dec767eea03d6d2d01e44ac37" exitCode=6
Jul 18 01:32:50.999769 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:50.999654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm" event={"ID":"a70ffe71-8bb4-4554-a038-6501f4fa22a0","Type":"ContainerDied","Data":"b98aaa7489932eba488eee72431637c924c7073dec767eea03d6d2d01e44ac37"}
Jul 18 01:32:50.999769 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:50.999748    2563 scope.go:117] "RemoveContainer" containerID="e6c4d6da5abfe46a490847ba30170fa0162d1070545fe5abc77fa5b2767f2734"
Jul 18 01:32:51.000121 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:51.000104    2563 scope.go:117] "RemoveContainer" containerID="b98aaa7489932eba488eee72431637c924c7073dec767eea03d6d2d01e44ac37"
Jul 18 01:32:51.000374 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:51.000354    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm_odh-ai-gateway-infra(a70ffe71-8bb4-4554-a038-6501f4fa22a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0"
Jul 18 01:32:52.024979 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:52.024933    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"]
Jul 18 01:32:52.166905 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:52.166878    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"
Jul 18 01:32:52.336670 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:52.336632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70ffe71-8bb4-4554-a038-6501f4fa22a0-maas-api-service-ca\") pod \"a70ffe71-8bb4-4554-a038-6501f4fa22a0\" (UID: \"a70ffe71-8bb4-4554-a038-6501f4fa22a0\") "
Jul 18 01:32:52.336991 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:52.336968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70ffe71-8bb4-4554-a038-6501f4fa22a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a70ffe71-8bb4-4554-a038-6501f4fa22a0" (UID: "a70ffe71-8bb4-4554-a038-6501f4fa22a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:32:52.437812 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:52.437771    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70ffe71-8bb4-4554-a038-6501f4fa22a0-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:32:53.014589 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.014560    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"
Jul 18 01:32:53.014801 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.014588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm" event={"ID":"a70ffe71-8bb4-4554-a038-6501f4fa22a0","Type":"ContainerDied","Data":"4391f63a4cfcb069b7fa0b0df023f85ac3ad46f5c9c5919019a8386efa9672ae"}
Jul 18 01:32:53.014801 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.014665    2563 scope.go:117] "RemoveContainer" containerID="b98aaa7489932eba488eee72431637c924c7073dec767eea03d6d2d01e44ac37"
Jul 18 01:32:53.036999 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.036967    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"]
Jul 18 01:32:53.042220 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.042190    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-f6qdm"]
Jul 18 01:32:53.405548 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405460    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjhbn"]
Jul 18 01:32:53.405877 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405864    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:53.405918 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405879    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd0ed86-ba54-49dd-93cb-14d26fbd4931" containerName="revoke-keys"
Jul 18 01:32:53.405918 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405891    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:32:53.405918 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405896    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:32:53.405918 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405905    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:32:53.405918 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405911    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:32:53.406109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405984    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:32:53.406109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.405991    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20345cc3-5abc-453b-8e70-6c475c88d8e2" containerName="revoke-keys"
Jul 18 01:32:53.406109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.406003    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:32:53.410519 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.410497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:32:53.417210 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.417173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjhbn"]
Jul 18 01:32:53.548165 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.548122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/8936d04c-7421-4b9f-846b-8d0dff4e0a3c-kube-api-access-k259n\") pod \"kuadrant-operator-catalog-xjhbn\" (UID: \"8936d04c-7421-4b9f-846b-8d0dff4e0a3c\") " pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:32:53.649379 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.649333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/8936d04c-7421-4b9f-846b-8d0dff4e0a3c-kube-api-access-k259n\") pod \"kuadrant-operator-catalog-xjhbn\" (UID: \"8936d04c-7421-4b9f-846b-8d0dff4e0a3c\") " pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:32:53.657493 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.657414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/8936d04c-7421-4b9f-846b-8d0dff4e0a3c-kube-api-access-k259n\") pod \"kuadrant-operator-catalog-xjhbn\" (UID: \"8936d04c-7421-4b9f-846b-8d0dff4e0a3c\") " pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:32:53.722477 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.722438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:32:53.872365 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:53.872330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjhbn"]
Jul 18 01:32:53.873706 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:32:53.873679    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8936d04c_7421_4b9f_846b_8d0dff4e0a3c.slice/crio-8c7b874a4cccdb7c3fe412f47238d006eb6d42de79d4c3a69eec1172831e6c87 WatchSource:0}: Error finding container 8c7b874a4cccdb7c3fe412f47238d006eb6d42de79d4c3a69eec1172831e6c87: Status 404 returned error can't find the container with id 8c7b874a4cccdb7c3fe412f47238d006eb6d42de79d4c3a69eec1172831e6c87
Jul 18 01:32:54.027279 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:54.027245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn" event={"ID":"8936d04c-7421-4b9f-846b-8d0dff4e0a3c","Type":"ContainerStarted","Data":"8c7b874a4cccdb7c3fe412f47238d006eb6d42de79d4c3a69eec1172831e6c87"}
Jul 18 01:32:54.176603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:54.176553    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" path="/var/lib/kubelet/pods/a70ffe71-8bb4-4554-a038-6501f4fa22a0/volumes"
Jul 18 01:32:55.035134 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:55.035090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn" event={"ID":"8936d04c-7421-4b9f-846b-8d0dff4e0a3c","Type":"ContainerStarted","Data":"a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec"}
Jul 18 01:32:55.051948 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:55.051891    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn" podStartSLOduration=1.555555618 podStartE2EDuration="2.051873876s" totalImagesPullingTime="496.318258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:32:55.048847338 +0000 UTC m=+3145.368824732" watchObservedRunningTime="2026-07-18 01:32:55.051873876 +0000 UTC m=+3145.371851304"
Jul 18 01:32:55.169443 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:55.169406    2563 scope.go:117] "RemoveContainer" containerID="95b4cdd2c16cf93385609abf8cd022a4373332d3c33a3f1407ace41488956384"
Jul 18 01:32:56.046679 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:56.046467    2563 generic.go:356] "Generic (PLEG): container finished" podID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerID="62beb4a66bc5e90d030116a49cdefabfaf2f74b9245d22e78a829fd4d259a2d6" exitCode=6
Jul 18 01:32:56.046679 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:56.046563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk" event={"ID":"e9051b65-4c29-4f2a-b8f8-d859a924f0d4","Type":"ContainerDied","Data":"62beb4a66bc5e90d030116a49cdefabfaf2f74b9245d22e78a829fd4d259a2d6"}
Jul 18 01:32:56.047214 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:56.046691    2563 scope.go:117] "RemoveContainer" containerID="95b4cdd2c16cf93385609abf8cd022a4373332d3c33a3f1407ace41488956384"
Jul 18 01:32:56.047665 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:56.047641    2563 scope.go:117] "RemoveContainer" containerID="62beb4a66bc5e90d030116a49cdefabfaf2f74b9245d22e78a829fd4d259a2d6"
Jul 18 01:32:56.047869 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:32:56.047840    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk_odh-ai-gateway-infra(e9051b65-4c29-4f2a-b8f8-d859a924f0d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4"
Jul 18 01:32:57.073141 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:57.073102    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"]
Jul 18 01:32:57.219975 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:57.219944    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"
Jul 18 01:32:57.386743 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:57.386639    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9051b65-4c29-4f2a-b8f8-d859a924f0d4-maas-api-service-ca\") pod \"e9051b65-4c29-4f2a-b8f8-d859a924f0d4\" (UID: \"e9051b65-4c29-4f2a-b8f8-d859a924f0d4\") "
Jul 18 01:32:57.387025 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:57.386998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9051b65-4c29-4f2a-b8f8-d859a924f0d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9051b65-4c29-4f2a-b8f8-d859a924f0d4" (UID: "e9051b65-4c29-4f2a-b8f8-d859a924f0d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:32:57.487574 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:57.487534    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9051b65-4c29-4f2a-b8f8-d859a924f0d4-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:32:58.061219 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:58.061186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"
Jul 18 01:32:58.061405 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:58.061227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk" event={"ID":"e9051b65-4c29-4f2a-b8f8-d859a924f0d4","Type":"ContainerDied","Data":"2468675b3d4b8f1204c8d7461e9ffd5e37edc33584727d5787fc4110a6f601f5"}
Jul 18 01:32:58.061405 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:58.061292    2563 scope.go:117] "RemoveContainer" containerID="62beb4a66bc5e90d030116a49cdefabfaf2f74b9245d22e78a829fd4d259a2d6"
Jul 18 01:32:58.085653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:58.085614    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"]
Jul 18 01:32:58.089397 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:58.089367    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-s2tzk"]
Jul 18 01:32:58.176578 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:32:58.176540    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" path="/var/lib/kubelet/pods/e9051b65-4c29-4f2a-b8f8-d859a924f0d4/volumes"
Jul 18 01:33:03.723495 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:03.723449    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:33:03.724134 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:03.723513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:33:03.746619 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:03.746581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:33:04.125390 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:04.125315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:33:04.157387 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:04.157351    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjhbn"]
Jul 18 01:33:06.117298 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:06.117241    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn" podUID="8936d04c-7421-4b9f-846b-8d0dff4e0a3c" containerName="registry-server" containerID="cri-o://a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec" gracePeriod=2
Jul 18 01:33:06.377977 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:06.377906    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:33:06.465991 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:06.465951    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/8936d04c-7421-4b9f-846b-8d0dff4e0a3c-kube-api-access-k259n\") pod \"8936d04c-7421-4b9f-846b-8d0dff4e0a3c\" (UID: \"8936d04c-7421-4b9f-846b-8d0dff4e0a3c\") "
Jul 18 01:33:06.468021 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:06.467994    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8936d04c-7421-4b9f-846b-8d0dff4e0a3c-kube-api-access-k259n" (OuterVolumeSpecName: "kube-api-access-k259n") pod "8936d04c-7421-4b9f-846b-8d0dff4e0a3c" (UID: "8936d04c-7421-4b9f-846b-8d0dff4e0a3c"). InnerVolumeSpecName "kube-api-access-k259n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:33:06.567418 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:06.567378    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k259n\" (UniqueName: \"kubernetes.io/projected/8936d04c-7421-4b9f-846b-8d0dff4e0a3c-kube-api-access-k259n\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:33:07.123883 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.123842    2563 generic.go:356] "Generic (PLEG): container finished" podID="8936d04c-7421-4b9f-846b-8d0dff4e0a3c" containerID="a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec" exitCode=0
Jul 18 01:33:07.123883 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.123888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn"
Jul 18 01:33:07.124450 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.123924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn" event={"ID":"8936d04c-7421-4b9f-846b-8d0dff4e0a3c","Type":"ContainerDied","Data":"a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec"}
Jul 18 01:33:07.124450 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.123978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xjhbn" event={"ID":"8936d04c-7421-4b9f-846b-8d0dff4e0a3c","Type":"ContainerDied","Data":"8c7b874a4cccdb7c3fe412f47238d006eb6d42de79d4c3a69eec1172831e6c87"}
Jul 18 01:33:07.124450 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.124004    2563 scope.go:117] "RemoveContainer" containerID="a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec"
Jul 18 01:33:07.133965 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.133945    2563 scope.go:117] "RemoveContainer" containerID="a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec"
Jul 18 01:33:07.134292 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:07.134273    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec\": container with ID starting with a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec not found: ID does not exist" containerID="a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec"
Jul 18 01:33:07.134357 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.134302    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec"} err="failed to get container status \"a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec\": rpc error: code = NotFound desc = could not find container \"a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec\": container with ID starting with a4e48de5dbcf38763f3d2de3118bc4f20be996179fd25ee12529954af1226aec not found: ID does not exist"
Jul 18 01:33:07.144799 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.144768    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjhbn"]
Jul 18 01:33:07.148701 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.148675    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjhbn"]
Jul 18 01:33:07.236522 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:07.236472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:33:08.176046 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:08.176010    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8936d04c-7421-4b9f-846b-8d0dff4e0a3c" path="/var/lib/kubelet/pods/8936d04c-7421-4b9f-846b-8d0dff4e0a3c/volumes"
Jul 18 01:33:09.374189 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374150    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"]
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374536    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374548    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374557    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374562    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374571    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374577    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374586    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374591    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374599    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8936d04c-7421-4b9f-846b-8d0dff4e0a3c" containerName="registry-server"
Jul 18 01:33:09.374610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374604    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8936d04c-7421-4b9f-846b-8d0dff4e0a3c" containerName="registry-server"
Jul 18 01:33:09.374953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374672    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374681    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374687    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70ffe71-8bb4-4554-a038-6501f4fa22a0" containerName="revoke-keys"
Jul 18 01:33:09.374953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374693    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9051b65-4c29-4f2a-b8f8-d859a924f0d4" containerName="revoke-keys"
Jul 18 01:33:09.374953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.374701    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8936d04c-7421-4b9f-846b-8d0dff4e0a3c" containerName="registry-server"
Jul 18 01:33:09.379367 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.379340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"
Jul 18 01:33:09.382488 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.382464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:33:09.385691 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.385663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"]
Jul 18 01:33:09.493801 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.493767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537f42f-07fb-44f6-93a2-03c207ef7e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj\" (UID: \"3537f42f-07fb-44f6-93a2-03c207ef7e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"
Jul 18 01:33:09.594897 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.594858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537f42f-07fb-44f6-93a2-03c207ef7e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj\" (UID: \"3537f42f-07fb-44f6-93a2-03c207ef7e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"
Jul 18 01:33:09.595532 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.595512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537f42f-07fb-44f6-93a2-03c207ef7e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj\" (UID: \"3537f42f-07fb-44f6-93a2-03c207ef7e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"
Jul 18 01:33:09.692841 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.692747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"
Jul 18 01:33:09.833358 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:09.833321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"]
Jul 18 01:33:09.834717 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:33:09.834685    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3537f42f_07fb_44f6_93a2_03c207ef7e41.slice/crio-ee01056fec758fdc8338f3d8967eab16f2e68a5d04706baadf5c2018f3c4aa88 WatchSource:0}: Error finding container ee01056fec758fdc8338f3d8967eab16f2e68a5d04706baadf5c2018f3c4aa88: Status 404 returned error can't find the container with id ee01056fec758fdc8338f3d8967eab16f2e68a5d04706baadf5c2018f3c4aa88
Jul 18 01:33:10.145322 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:10.145278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj" event={"ID":"3537f42f-07fb-44f6-93a2-03c207ef7e41","Type":"ContainerStarted","Data":"ee01056fec758fdc8338f3d8967eab16f2e68a5d04706baadf5c2018f3c4aa88"}
Jul 18 01:33:11.152825 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:11.152788    2563 generic.go:356] "Generic (PLEG): container finished" podID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerID="fdbfe12b17b77234507a3c859b9d2385bf66c0d673f6393ed858a710e797c10a" exitCode=6
Jul 18 01:33:11.153232 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:11.152850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj" event={"ID":"3537f42f-07fb-44f6-93a2-03c207ef7e41","Type":"ContainerDied","Data":"fdbfe12b17b77234507a3c859b9d2385bf66c0d673f6393ed858a710e797c10a"}
Jul 18 01:33:11.153232 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:11.153086    2563 scope.go:117] "RemoveContainer" containerID="fdbfe12b17b77234507a3c859b9d2385bf66c0d673f6393ed858a710e797c10a"
Jul 18 01:33:12.160601 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:12.160495    2563 generic.go:356] "Generic (PLEG): container finished" podID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerID="8e57428862adb63a7f130266fb658712e0c1ff2b59e2842581eeb619f43970f1" exitCode=6
Jul 18 01:33:12.160601 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:12.160548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj" event={"ID":"3537f42f-07fb-44f6-93a2-03c207ef7e41","Type":"ContainerDied","Data":"8e57428862adb63a7f130266fb658712e0c1ff2b59e2842581eeb619f43970f1"}
Jul 18 01:33:12.161142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:12.160622    2563 scope.go:117] "RemoveContainer" containerID="fdbfe12b17b77234507a3c859b9d2385bf66c0d673f6393ed858a710e797c10a"
Jul 18 01:33:12.161142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:12.160821    2563 scope.go:117] "RemoveContainer" containerID="8e57428862adb63a7f130266fb658712e0c1ff2b59e2842581eeb619f43970f1"
Jul 18 01:33:12.161142 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:12.161029    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj_odh-ai-gateway-infra(3537f42f-07fb-44f6-93a2-03c207ef7e41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41"
Jul 18 01:33:13.171488 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:13.171456    2563 scope.go:117] "RemoveContainer" containerID="8e57428862adb63a7f130266fb658712e0c1ff2b59e2842581eeb619f43970f1"
Jul 18 01:33:13.171959 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:13.171716    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj_odh-ai-gateway-infra(3537f42f-07fb-44f6-93a2-03c207ef7e41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41"
Jul 18 01:33:15.424921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:15.424881    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"]
Jul 18 01:33:15.429756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:15.429736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"
Jul 18 01:33:15.436296 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:15.436265    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"]
Jul 18 01:33:15.549434 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:15.549393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/491dfbd9-e71e-48de-b35c-ec4f9af97ef2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-w26pm\" (UID: \"491dfbd9-e71e-48de-b35c-ec4f9af97ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"
Jul 18 01:33:15.650097 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:15.650024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/491dfbd9-e71e-48de-b35c-ec4f9af97ef2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-w26pm\" (UID: \"491dfbd9-e71e-48de-b35c-ec4f9af97ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"
Jul 18 01:33:15.650681 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:15.650661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/491dfbd9-e71e-48de-b35c-ec4f9af97ef2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-w26pm\" (UID: \"491dfbd9-e71e-48de-b35c-ec4f9af97ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"
Jul 18 01:33:15.741170 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:15.741136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"
Jul 18 01:33:15.883497 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:33:15.883468    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491dfbd9_e71e_48de_b35c_ec4f9af97ef2.slice/crio-142774bc42fffe7b8dde2b580c650df664deadb43eae593ae675ec7389658fc1 WatchSource:0}: Error finding container 142774bc42fffe7b8dde2b580c650df664deadb43eae593ae675ec7389658fc1: Status 404 returned error can't find the container with id 142774bc42fffe7b8dde2b580c650df664deadb43eae593ae675ec7389658fc1
Jul 18 01:33:15.883759 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:15.883732    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"]
Jul 18 01:33:16.191262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:16.191232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm" event={"ID":"491dfbd9-e71e-48de-b35c-ec4f9af97ef2","Type":"ContainerStarted","Data":"142774bc42fffe7b8dde2b580c650df664deadb43eae593ae675ec7389658fc1"}
Jul 18 01:33:17.198699 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:17.198658    2563 generic.go:356] "Generic (PLEG): container finished" podID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerID="1a25f78ca48231dc4dc320481cb4048d1ccb834395fd64104a053a021416ed66" exitCode=6
Jul 18 01:33:17.199115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:17.198744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm" event={"ID":"491dfbd9-e71e-48de-b35c-ec4f9af97ef2","Type":"ContainerDied","Data":"1a25f78ca48231dc4dc320481cb4048d1ccb834395fd64104a053a021416ed66"}
Jul 18 01:33:17.199115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:17.198937    2563 scope.go:117] "RemoveContainer" containerID="1a25f78ca48231dc4dc320481cb4048d1ccb834395fd64104a053a021416ed66"
Jul 18 01:33:18.206886 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:18.206852    2563 generic.go:356] "Generic (PLEG): container finished" podID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerID="b115e4fd29e3d0c263f50d56770391fe01e5a9032653cb5f7e5474a9452d954c" exitCode=6
Jul 18 01:33:18.207351 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:18.206936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm" event={"ID":"491dfbd9-e71e-48de-b35c-ec4f9af97ef2","Type":"ContainerDied","Data":"b115e4fd29e3d0c263f50d56770391fe01e5a9032653cb5f7e5474a9452d954c"}
Jul 18 01:33:18.207351 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:18.207005    2563 scope.go:117] "RemoveContainer" containerID="1a25f78ca48231dc4dc320481cb4048d1ccb834395fd64104a053a021416ed66"
Jul 18 01:33:18.207351 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:18.207222    2563 scope.go:117] "RemoveContainer" containerID="b115e4fd29e3d0c263f50d56770391fe01e5a9032653cb5f7e5474a9452d954c"
Jul 18 01:33:18.207495 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:18.207419    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-w26pm_odh-ai-gateway-infra(491dfbd9-e71e-48de-b35c-ec4f9af97ef2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2"
Jul 18 01:33:19.215095 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:19.215041    2563 scope.go:117] "RemoveContainer" containerID="b115e4fd29e3d0c263f50d56770391fe01e5a9032653cb5f7e5474a9452d954c"
Jul 18 01:33:19.215652 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:19.215290    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-w26pm_odh-ai-gateway-infra(491dfbd9-e71e-48de-b35c-ec4f9af97ef2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2"
Jul 18 01:33:25.738911 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:25.738870    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:33:27.169517 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:27.169477    2563 scope.go:117] "RemoveContainer" containerID="8e57428862adb63a7f130266fb658712e0c1ff2b59e2842581eeb619f43970f1"
Jul 18 01:33:28.274376 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:28.274342    2563 generic.go:356] "Generic (PLEG): container finished" podID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerID="1f6a1ead7f4adbb4b1a6214136c996cb315c0f8d0224a54bb75fbbe358d3e189" exitCode=6
Jul 18 01:33:28.274878 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:28.274421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj" event={"ID":"3537f42f-07fb-44f6-93a2-03c207ef7e41","Type":"ContainerDied","Data":"1f6a1ead7f4adbb4b1a6214136c996cb315c0f8d0224a54bb75fbbe358d3e189"}
Jul 18 01:33:28.274878 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:28.274488    2563 scope.go:117] "RemoveContainer" containerID="8e57428862adb63a7f130266fb658712e0c1ff2b59e2842581eeb619f43970f1"
Jul 18 01:33:28.274878 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:28.274847    2563 scope.go:117] "RemoveContainer" containerID="1f6a1ead7f4adbb4b1a6214136c996cb315c0f8d0224a54bb75fbbe358d3e189"
Jul 18 01:33:28.275132 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:28.275109    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj_odh-ai-gateway-infra(3537f42f-07fb-44f6-93a2-03c207ef7e41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41"
Jul 18 01:33:29.303653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:29.303612    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"]
Jul 18 01:33:29.450176 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:29.450147    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"
Jul 18 01:33:29.573701 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:29.573606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537f42f-07fb-44f6-93a2-03c207ef7e41-maas-api-service-ca\") pod \"3537f42f-07fb-44f6-93a2-03c207ef7e41\" (UID: \"3537f42f-07fb-44f6-93a2-03c207ef7e41\") "
Jul 18 01:33:29.573995 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:29.573967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3537f42f-07fb-44f6-93a2-03c207ef7e41-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3537f42f-07fb-44f6-93a2-03c207ef7e41" (UID: "3537f42f-07fb-44f6-93a2-03c207ef7e41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:33:29.674734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:29.674693    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537f42f-07fb-44f6-93a2-03c207ef7e41-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:33:30.291399 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:30.291371    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"
Jul 18 01:33:30.291582 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:30.291369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj" event={"ID":"3537f42f-07fb-44f6-93a2-03c207ef7e41","Type":"ContainerDied","Data":"ee01056fec758fdc8338f3d8967eab16f2e68a5d04706baadf5c2018f3c4aa88"}
Jul 18 01:33:30.291582 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:30.291511    2563 scope.go:117] "RemoveContainer" containerID="1f6a1ead7f4adbb4b1a6214136c996cb315c0f8d0224a54bb75fbbe358d3e189"
Jul 18 01:33:30.308421 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:30.308388    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"]
Jul 18 01:33:30.322759 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:30.322726    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-rw4rj"]
Jul 18 01:33:31.170395 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:31.170362    2563 scope.go:117] "RemoveContainer" containerID="b115e4fd29e3d0c263f50d56770391fe01e5a9032653cb5f7e5474a9452d954c"
Jul 18 01:33:32.175931 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.175845    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" path="/var/lib/kubelet/pods/3537f42f-07fb-44f6-93a2-03c207ef7e41/volumes"
Jul 18 01:33:32.305440 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.305403    2563 generic.go:356] "Generic (PLEG): container finished" podID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerID="7fb002060892286983d32305d6f9e46b6507eda334dbcdc0084ed6b917cf4822" exitCode=6
Jul 18 01:33:32.305645 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.305474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm" event={"ID":"491dfbd9-e71e-48de-b35c-ec4f9af97ef2","Type":"ContainerDied","Data":"7fb002060892286983d32305d6f9e46b6507eda334dbcdc0084ed6b917cf4822"}
Jul 18 01:33:32.305645 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.305540    2563 scope.go:117] "RemoveContainer" containerID="b115e4fd29e3d0c263f50d56770391fe01e5a9032653cb5f7e5474a9452d954c"
Jul 18 01:33:32.305857 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.305842    2563 scope.go:117] "RemoveContainer" containerID="7fb002060892286983d32305d6f9e46b6507eda334dbcdc0084ed6b917cf4822"
Jul 18 01:33:32.306071 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:32.306027    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-w26pm_odh-ai-gateway-infra(491dfbd9-e71e-48de-b35c-ec4f9af97ef2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2"
Jul 18 01:33:32.511255 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.511215    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"]
Jul 18 01:33:32.511642 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.511629    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:32.511692 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.511644    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:32.511692 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.511660    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:32.511692 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.511666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:32.511787 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.511727    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:32.511787 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.511737    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:32.516179 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.516150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"
Jul 18 01:33:32.522264 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.522231    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"]
Jul 18 01:33:32.601971 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.601927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f2f524-dee0-40e9-9006-b85586c9e1a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r\" (UID: \"86f2f524-dee0-40e9-9006-b85586c9e1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"
Jul 18 01:33:32.703213 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.703172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f2f524-dee0-40e9-9006-b85586c9e1a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r\" (UID: \"86f2f524-dee0-40e9-9006-b85586c9e1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"
Jul 18 01:33:32.703820 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.703799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f2f524-dee0-40e9-9006-b85586c9e1a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r\" (UID: \"86f2f524-dee0-40e9-9006-b85586c9e1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"
Jul 18 01:33:32.828451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.828349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"
Jul 18 01:33:32.967556 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:32.967525    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"]
Jul 18 01:33:32.968693 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:33:32.968664    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f2f524_dee0_40e9_9006_b85586c9e1a2.slice/crio-a2e5943aa430089ca588188f8beb5f1a41be71f7038071e57c25053e402dabf6 WatchSource:0}: Error finding container a2e5943aa430089ca588188f8beb5f1a41be71f7038071e57c25053e402dabf6: Status 404 returned error can't find the container with id a2e5943aa430089ca588188f8beb5f1a41be71f7038071e57c25053e402dabf6
Jul 18 01:33:33.316297 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:33.316257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" event={"ID":"86f2f524-dee0-40e9-9006-b85586c9e1a2","Type":"ContainerStarted","Data":"ae0e9cb75b7de305d3e30aa80dbca04da677d4d2866e23e505db6d4a66674092"}
Jul 18 01:33:33.316691 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:33.316326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" event={"ID":"86f2f524-dee0-40e9-9006-b85586c9e1a2","Type":"ContainerStarted","Data":"a2e5943aa430089ca588188f8beb5f1a41be71f7038071e57c25053e402dabf6"}
Jul 18 01:33:33.332078 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:33.332025    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"]
Jul 18 01:33:33.334482 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:33.334437    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" podStartSLOduration=1.33442188 podStartE2EDuration="1.33442188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:33:33.331378059 +0000 UTC m=+3183.651355467" watchObservedRunningTime="2026-07-18 01:33:33.33442188 +0000 UTC m=+3183.654399281"
Jul 18 01:33:33.466466 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:33.466437    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"
Jul 18 01:33:33.611583 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:33.611549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/491dfbd9-e71e-48de-b35c-ec4f9af97ef2-maas-api-service-ca\") pod \"491dfbd9-e71e-48de-b35c-ec4f9af97ef2\" (UID: \"491dfbd9-e71e-48de-b35c-ec4f9af97ef2\") "
Jul 18 01:33:33.611919 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:33.611894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491dfbd9-e71e-48de-b35c-ec4f9af97ef2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "491dfbd9-e71e-48de-b35c-ec4f9af97ef2" (UID: "491dfbd9-e71e-48de-b35c-ec4f9af97ef2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:33:33.712185 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:33.712147    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/491dfbd9-e71e-48de-b35c-ec4f9af97ef2-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:33:34.323185 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:34.323148    2563 generic.go:356] "Generic (PLEG): container finished" podID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerID="ae0e9cb75b7de305d3e30aa80dbca04da677d4d2866e23e505db6d4a66674092" exitCode=6
Jul 18 01:33:34.323642 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:34.323229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" event={"ID":"86f2f524-dee0-40e9-9006-b85586c9e1a2","Type":"ContainerDied","Data":"ae0e9cb75b7de305d3e30aa80dbca04da677d4d2866e23e505db6d4a66674092"}
Jul 18 01:33:34.323642 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:34.323365    2563 scope.go:117] "RemoveContainer" containerID="ae0e9cb75b7de305d3e30aa80dbca04da677d4d2866e23e505db6d4a66674092"
Jul 18 01:33:34.324700 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:34.324680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm" event={"ID":"491dfbd9-e71e-48de-b35c-ec4f9af97ef2","Type":"ContainerDied","Data":"142774bc42fffe7b8dde2b580c650df664deadb43eae593ae675ec7389658fc1"}
Jul 18 01:33:34.324803 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:34.324720    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"
Jul 18 01:33:34.324803 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:34.324729    2563 scope.go:117] "RemoveContainer" containerID="7fb002060892286983d32305d6f9e46b6507eda334dbcdc0084ed6b917cf4822"
Jul 18 01:33:34.355887 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:34.355850    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"]
Jul 18 01:33:34.358430 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:34.358407    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-w26pm"]
Jul 18 01:33:35.332597 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:35.332560    2563 generic.go:356] "Generic (PLEG): container finished" podID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerID="7e9a02036b697126bd71057e221fe3c8e0688938b7be409f77fe707957e76754" exitCode=6
Jul 18 01:33:35.333032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:35.332645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" event={"ID":"86f2f524-dee0-40e9-9006-b85586c9e1a2","Type":"ContainerDied","Data":"7e9a02036b697126bd71057e221fe3c8e0688938b7be409f77fe707957e76754"}
Jul 18 01:33:35.333032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:35.332705    2563 scope.go:117] "RemoveContainer" containerID="ae0e9cb75b7de305d3e30aa80dbca04da677d4d2866e23e505db6d4a66674092"
Jul 18 01:33:35.333032 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:35.332932    2563 scope.go:117] "RemoveContainer" containerID="7e9a02036b697126bd71057e221fe3c8e0688938b7be409f77fe707957e76754"
Jul 18 01:33:35.333231 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:35.333204    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r_odh-ai-gateway-infra(86f2f524-dee0-40e9-9006-b85586c9e1a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2"
Jul 18 01:33:36.175770 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:36.175736    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" path="/var/lib/kubelet/pods/491dfbd9-e71e-48de-b35c-ec4f9af97ef2/volumes"
Jul 18 01:33:36.345326 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:36.345297    2563 scope.go:117] "RemoveContainer" containerID="7e9a02036b697126bd71057e221fe3c8e0688938b7be409f77fe707957e76754"
Jul 18 01:33:36.345676 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:36.345498    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r_odh-ai-gateway-infra(86f2f524-dee0-40e9-9006-b85586c9e1a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2"
Jul 18 01:33:37.449091 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449045    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"]
Jul 18 01:33:37.449521 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449507    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:37.449569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449523    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:37.449569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449539    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:33:37.449569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449545    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:33:37.449569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449551    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:33:37.449569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449558    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:33:37.449726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449574    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:33:37.449726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449581    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:33:37.449726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449671    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:33:37.449726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449681    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:33:37.449726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.449690    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3537f42f-07fb-44f6-93a2-03c207ef7e41" containerName="revoke-keys"
Jul 18 01:33:37.454228 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.454210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"
Jul 18 01:33:37.459675 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.459650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"]
Jul 18 01:33:37.544501 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.544455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4938da-000c-4da6-a66e-7bc5e70ec173-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq\" (UID: \"0e4938da-000c-4da6-a66e-7bc5e70ec173\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"
Jul 18 01:33:37.645693 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.645650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4938da-000c-4da6-a66e-7bc5e70ec173-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq\" (UID: \"0e4938da-000c-4da6-a66e-7bc5e70ec173\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"
Jul 18 01:33:37.646325 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.646308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4938da-000c-4da6-a66e-7bc5e70ec173-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq\" (UID: \"0e4938da-000c-4da6-a66e-7bc5e70ec173\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"
Jul 18 01:33:37.766767 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.766675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"
Jul 18 01:33:37.908102 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:37.908067    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"]
Jul 18 01:33:37.909770 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:33:37.909743    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4938da_000c_4da6_a66e_7bc5e70ec173.slice/crio-3d206b29a88f8b3fffd575602430d205197247498f714aae963a965ea5e7f329 WatchSource:0}: Error finding container 3d206b29a88f8b3fffd575602430d205197247498f714aae963a965ea5e7f329: Status 404 returned error can't find the container with id 3d206b29a88f8b3fffd575602430d205197247498f714aae963a965ea5e7f329
Jul 18 01:33:38.359014 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:38.358918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" event={"ID":"0e4938da-000c-4da6-a66e-7bc5e70ec173","Type":"ContainerStarted","Data":"d8aea49d2301db251fb15c4b624bbddb146461ddfd74858d6bf83aea4e858bae"}
Jul 18 01:33:38.359014 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:38.358985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" event={"ID":"0e4938da-000c-4da6-a66e-7bc5e70ec173","Type":"ContainerStarted","Data":"3d206b29a88f8b3fffd575602430d205197247498f714aae963a965ea5e7f329"}
Jul 18 01:33:38.374692 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:38.374639    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" podStartSLOduration=1.374621162 podStartE2EDuration="1.374621162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:33:38.374253016 +0000 UTC m=+3188.694230400" watchObservedRunningTime="2026-07-18 01:33:38.374621162 +0000 UTC m=+3188.694598592"
Jul 18 01:33:39.367299 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:39.367260    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerID="d8aea49d2301db251fb15c4b624bbddb146461ddfd74858d6bf83aea4e858bae" exitCode=6
Jul 18 01:33:39.367685 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:39.367348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" event={"ID":"0e4938da-000c-4da6-a66e-7bc5e70ec173","Type":"ContainerDied","Data":"d8aea49d2301db251fb15c4b624bbddb146461ddfd74858d6bf83aea4e858bae"}
Jul 18 01:33:39.367685 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:39.367533    2563 scope.go:117] "RemoveContainer" containerID="d8aea49d2301db251fb15c4b624bbddb146461ddfd74858d6bf83aea4e858bae"
Jul 18 01:33:40.375768 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:40.375670    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerID="88272c187c391920bd5dc43ba6fd0e76778ac24971a4a0eeb25770e69c8d9d1a" exitCode=6
Jul 18 01:33:40.376249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:40.375760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" event={"ID":"0e4938da-000c-4da6-a66e-7bc5e70ec173","Type":"ContainerDied","Data":"88272c187c391920bd5dc43ba6fd0e76778ac24971a4a0eeb25770e69c8d9d1a"}
Jul 18 01:33:40.376249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:40.375818    2563 scope.go:117] "RemoveContainer" containerID="d8aea49d2301db251fb15c4b624bbddb146461ddfd74858d6bf83aea4e858bae"
Jul 18 01:33:40.376249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:40.376131    2563 scope.go:117] "RemoveContainer" containerID="88272c187c391920bd5dc43ba6fd0e76778ac24971a4a0eeb25770e69c8d9d1a"
Jul 18 01:33:40.376368 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:40.376340    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq_odh-ai-gateway-infra(0e4938da-000c-4da6-a66e-7bc5e70ec173)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173"
Jul 18 01:33:41.384282 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:41.384251    2563 scope.go:117] "RemoveContainer" containerID="88272c187c391920bd5dc43ba6fd0e76778ac24971a4a0eeb25770e69c8d9d1a"
Jul 18 01:33:41.384650 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:41.384430    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq_odh-ai-gateway-infra(0e4938da-000c-4da6-a66e-7bc5e70ec173)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173"
Jul 18 01:33:43.538541 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:43.538505    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:33:51.169784 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:51.169751    2563 scope.go:117] "RemoveContainer" containerID="7e9a02036b697126bd71057e221fe3c8e0688938b7be409f77fe707957e76754"
Jul 18 01:33:51.455445 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:51.455410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" event={"ID":"86f2f524-dee0-40e9-9006-b85586c9e1a2","Type":"ContainerStarted","Data":"e6e1486159ff4e903928eb02e72131be80a6992b37abf83c384cbbf4f50d7fad"}
Jul 18 01:33:52.170300 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:52.170212    2563 scope.go:117] "RemoveContainer" containerID="88272c187c391920bd5dc43ba6fd0e76778ac24971a4a0eeb25770e69c8d9d1a"
Jul 18 01:33:52.463731 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:52.463700    2563 generic.go:356] "Generic (PLEG): container finished" podID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerID="e6e1486159ff4e903928eb02e72131be80a6992b37abf83c384cbbf4f50d7fad" exitCode=6
Jul 18 01:33:52.463912 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:52.463776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" event={"ID":"86f2f524-dee0-40e9-9006-b85586c9e1a2","Type":"ContainerDied","Data":"e6e1486159ff4e903928eb02e72131be80a6992b37abf83c384cbbf4f50d7fad"}
Jul 18 01:33:52.463912 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:52.463837    2563 scope.go:117] "RemoveContainer" containerID="7e9a02036b697126bd71057e221fe3c8e0688938b7be409f77fe707957e76754"
Jul 18 01:33:52.464135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:52.464118    2563 scope.go:117] "RemoveContainer" containerID="e6e1486159ff4e903928eb02e72131be80a6992b37abf83c384cbbf4f50d7fad"
Jul 18 01:33:52.464379 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:52.464355    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r_odh-ai-gateway-infra(86f2f524-dee0-40e9-9006-b85586c9e1a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2"
Jul 18 01:33:52.488626 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:52.488600    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"]
Jul 18 01:33:53.471715 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:53.471680    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerID="d4367217ef193e85be5fe1dac38f030ede8e6438e1a872863e16890de1f1e628" exitCode=6
Jul 18 01:33:53.472209 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:53.471755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" event={"ID":"0e4938da-000c-4da6-a66e-7bc5e70ec173","Type":"ContainerDied","Data":"d4367217ef193e85be5fe1dac38f030ede8e6438e1a872863e16890de1f1e628"}
Jul 18 01:33:53.472209 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:53.471833    2563 scope.go:117] "RemoveContainer" containerID="88272c187c391920bd5dc43ba6fd0e76778ac24971a4a0eeb25770e69c8d9d1a"
Jul 18 01:33:53.472209 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:53.472198    2563 scope.go:117] "RemoveContainer" containerID="d4367217ef193e85be5fe1dac38f030ede8e6438e1a872863e16890de1f1e628"
Jul 18 01:33:53.472454 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:33:53.472430    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq_odh-ai-gateway-infra(0e4938da-000c-4da6-a66e-7bc5e70ec173)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173"
Jul 18 01:33:53.620023 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:53.619995    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"
Jul 18 01:33:53.795565 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:53.795469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f2f524-dee0-40e9-9006-b85586c9e1a2-maas-api-service-ca\") pod \"86f2f524-dee0-40e9-9006-b85586c9e1a2\" (UID: \"86f2f524-dee0-40e9-9006-b85586c9e1a2\") "
Jul 18 01:33:53.795847 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:53.795819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f2f524-dee0-40e9-9006-b85586c9e1a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86f2f524-dee0-40e9-9006-b85586c9e1a2" (UID: "86f2f524-dee0-40e9-9006-b85586c9e1a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:33:53.896597 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:53.896549    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f2f524-dee0-40e9-9006-b85586c9e1a2-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:33:54.483099 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.483041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r" event={"ID":"86f2f524-dee0-40e9-9006-b85586c9e1a2","Type":"ContainerDied","Data":"a2e5943aa430089ca588188f8beb5f1a41be71f7038071e57c25053e402dabf6"}
Jul 18 01:33:54.483571 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.483124    2563 scope.go:117] "RemoveContainer" containerID="e6e1486159ff4e903928eb02e72131be80a6992b37abf83c384cbbf4f50d7fad"
Jul 18 01:33:54.483571 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.483081    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"
Jul 18 01:33:54.508674 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.508636    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"]
Jul 18 01:33:54.538942 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.538903    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"]
Jul 18 01:33:54.543843 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.543810    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-p2q4r"]
Jul 18 01:33:54.651805 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.651778    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"
Jul 18 01:33:54.704308 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.704271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4938da-000c-4da6-a66e-7bc5e70ec173-maas-api-service-ca\") pod \"0e4938da-000c-4da6-a66e-7bc5e70ec173\" (UID: \"0e4938da-000c-4da6-a66e-7bc5e70ec173\") "
Jul 18 01:33:54.704644 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.704619    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4938da-000c-4da6-a66e-7bc5e70ec173-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e4938da-000c-4da6-a66e-7bc5e70ec173" (UID: "0e4938da-000c-4da6-a66e-7bc5e70ec173"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:33:54.804930 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:54.804832    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e4938da-000c-4da6-a66e-7bc5e70ec173-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:33:55.490817 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:55.490776    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"
Jul 18 01:33:55.491329 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:55.490783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq" event={"ID":"0e4938da-000c-4da6-a66e-7bc5e70ec173","Type":"ContainerDied","Data":"3d206b29a88f8b3fffd575602430d205197247498f714aae963a965ea5e7f329"}
Jul 18 01:33:55.491329 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:55.490921    2563 scope.go:117] "RemoveContainer" containerID="d4367217ef193e85be5fe1dac38f030ede8e6438e1a872863e16890de1f1e628"
Jul 18 01:33:55.516219 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:55.516186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"]
Jul 18 01:33:55.521601 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:55.521573    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-4wgqq"]
Jul 18 01:33:56.181383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:56.181347    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" path="/var/lib/kubelet/pods/0e4938da-000c-4da6-a66e-7bc5e70ec173/volumes"
Jul 18 01:33:56.181726 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:33:56.181712    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" path="/var/lib/kubelet/pods/86f2f524-dee0-40e9-9006-b85586c9e1a2/volumes"
Jul 18 01:34:01.847615 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:01.847573    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:34:05.461776 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.461739    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"]
Jul 18 01:34:05.462363 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462342    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462365    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462378    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462388    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462406    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462415    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462434    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.462451 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462443    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462456    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462465    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462484    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462492    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462579    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462592    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462603    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462612    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462623    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="491dfbd9-e71e-48de-b35c-ec4f9af97ef2" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462633    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f2f524-dee0-40e9-9006-b85586c9e1a2" containerName="revoke-keys"
Jul 18 01:34:05.462786 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.462643    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4938da-000c-4da6-a66e-7bc5e70ec173" containerName="revoke-keys"
Jul 18 01:34:05.467364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.467340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"
Jul 18 01:34:05.470266 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.469999    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:34:05.472109 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.472086    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"]
Jul 18 01:34:05.488405 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.488369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8d14e4-5614-410f-becc-e66da26c9bfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-qwtbx\" (UID: \"cb8d14e4-5614-410f-becc-e66da26c9bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"
Jul 18 01:34:05.589737 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.589689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8d14e4-5614-410f-becc-e66da26c9bfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-qwtbx\" (UID: \"cb8d14e4-5614-410f-becc-e66da26c9bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"
Jul 18 01:34:05.590371 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.590351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8d14e4-5614-410f-becc-e66da26c9bfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-qwtbx\" (UID: \"cb8d14e4-5614-410f-becc-e66da26c9bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"
Jul 18 01:34:05.780887 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.780792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"
Jul 18 01:34:05.924367 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:05.924335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"]
Jul 18 01:34:05.926277 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:34:05.926251    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8d14e4_5614_410f_becc_e66da26c9bfa.slice/crio-a7bb02dc434d7057805ba3d2474c8fa139e6cc2a6be87df753c4b9c8e0679bb3 WatchSource:0}: Error finding container a7bb02dc434d7057805ba3d2474c8fa139e6cc2a6be87df753c4b9c8e0679bb3: Status 404 returned error can't find the container with id a7bb02dc434d7057805ba3d2474c8fa139e6cc2a6be87df753c4b9c8e0679bb3
Jul 18 01:34:06.559948 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:06.559908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" event={"ID":"cb8d14e4-5614-410f-becc-e66da26c9bfa","Type":"ContainerStarted","Data":"08df19c596673bcc984779340e8a8e647c51cf5b289c272695905f002b6ae4df"}
Jul 18 01:34:06.560414 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:06.559964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" event={"ID":"cb8d14e4-5614-410f-becc-e66da26c9bfa","Type":"ContainerStarted","Data":"a7bb02dc434d7057805ba3d2474c8fa139e6cc2a6be87df753c4b9c8e0679bb3"}
Jul 18 01:34:06.575909 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:06.575858    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" podStartSLOduration=1.5758430460000001 podStartE2EDuration="1.575843046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:34:06.574996149 +0000 UTC m=+3216.894973525" watchObservedRunningTime="2026-07-18 01:34:06.575843046 +0000 UTC m=+3216.895820429"
Jul 18 01:34:07.568476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:07.568435    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerID="08df19c596673bcc984779340e8a8e647c51cf5b289c272695905f002b6ae4df" exitCode=6
Jul 18 01:34:07.568891 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:07.568528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" event={"ID":"cb8d14e4-5614-410f-becc-e66da26c9bfa","Type":"ContainerDied","Data":"08df19c596673bcc984779340e8a8e647c51cf5b289c272695905f002b6ae4df"}
Jul 18 01:34:07.568891 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:07.568739    2563 scope.go:117] "RemoveContainer" containerID="08df19c596673bcc984779340e8a8e647c51cf5b289c272695905f002b6ae4df"
Jul 18 01:34:08.576364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:08.576276    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerID="bc81e667bbe55703882e8060de83fc6a3b4aa96025d8dce6136c0841cd5d01ab" exitCode=6
Jul 18 01:34:08.576763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:08.576355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" event={"ID":"cb8d14e4-5614-410f-becc-e66da26c9bfa","Type":"ContainerDied","Data":"bc81e667bbe55703882e8060de83fc6a3b4aa96025d8dce6136c0841cd5d01ab"}
Jul 18 01:34:08.576763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:08.576421    2563 scope.go:117] "RemoveContainer" containerID="08df19c596673bcc984779340e8a8e647c51cf5b289c272695905f002b6ae4df"
Jul 18 01:34:08.576763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:08.576706    2563 scope.go:117] "RemoveContainer" containerID="bc81e667bbe55703882e8060de83fc6a3b4aa96025d8dce6136c0841cd5d01ab"
Jul 18 01:34:08.576935 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:08.576913    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-qwtbx_odh-ai-gateway-infra(cb8d14e4-5614-410f-becc-e66da26c9bfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa"
Jul 18 01:34:09.411620 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.411583    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"]
Jul 18 01:34:09.416362 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.416337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"
Jul 18 01:34:09.421352 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.421314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc77547-44bd-4735-aef4-591c3160411a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj\" (UID: \"5dc77547-44bd-4735-aef4-591c3160411a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"
Jul 18 01:34:09.422279 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.422248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"]
Jul 18 01:34:09.522497 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.522450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc77547-44bd-4735-aef4-591c3160411a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj\" (UID: \"5dc77547-44bd-4735-aef4-591c3160411a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"
Jul 18 01:34:09.523224 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.523199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc77547-44bd-4735-aef4-591c3160411a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj\" (UID: \"5dc77547-44bd-4735-aef4-591c3160411a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"
Jul 18 01:34:09.585213 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.585181    2563 scope.go:117] "RemoveContainer" containerID="bc81e667bbe55703882e8060de83fc6a3b4aa96025d8dce6136c0841cd5d01ab"
Jul 18 01:34:09.585626 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:09.585404    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-qwtbx_odh-ai-gateway-infra(cb8d14e4-5614-410f-becc-e66da26c9bfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa"
Jul 18 01:34:09.729082 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.729022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"
Jul 18 01:34:09.870839 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:09.870811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"]
Jul 18 01:34:09.872005 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:34:09.871976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc77547_44bd_4735_aef4_591c3160411a.slice/crio-16a288e58404f4e5bbed3c0cc1c6bf45999a4c6fb160a204ad007dbd0562dbc5 WatchSource:0}: Error finding container 16a288e58404f4e5bbed3c0cc1c6bf45999a4c6fb160a204ad007dbd0562dbc5: Status 404 returned error can't find the container with id 16a288e58404f4e5bbed3c0cc1c6bf45999a4c6fb160a204ad007dbd0562dbc5
Jul 18 01:34:10.592172 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:10.592134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" event={"ID":"5dc77547-44bd-4735-aef4-591c3160411a","Type":"ContainerStarted","Data":"35e8401e15d2a5d22885d79469373775152a664d9407931968c564dccd763f08"}
Jul 18 01:34:10.592646 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:10.592197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" event={"ID":"5dc77547-44bd-4735-aef4-591c3160411a","Type":"ContainerStarted","Data":"16a288e58404f4e5bbed3c0cc1c6bf45999a4c6fb160a204ad007dbd0562dbc5"}
Jul 18 01:34:10.609849 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:10.609785    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" podStartSLOduration=1.609768254 podStartE2EDuration="1.609768254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:34:10.607796801 +0000 UTC m=+3220.927774185" watchObservedRunningTime="2026-07-18 01:34:10.609768254 +0000 UTC m=+3220.929745637"
Jul 18 01:34:11.599504 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:11.599463    2563 generic.go:356] "Generic (PLEG): container finished" podID="5dc77547-44bd-4735-aef4-591c3160411a" containerID="35e8401e15d2a5d22885d79469373775152a664d9407931968c564dccd763f08" exitCode=6
Jul 18 01:34:11.599900 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:11.599525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" event={"ID":"5dc77547-44bd-4735-aef4-591c3160411a","Type":"ContainerDied","Data":"35e8401e15d2a5d22885d79469373775152a664d9407931968c564dccd763f08"}
Jul 18 01:34:11.599900 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:11.599847    2563 scope.go:117] "RemoveContainer" containerID="35e8401e15d2a5d22885d79469373775152a664d9407931968c564dccd763f08"
Jul 18 01:34:12.607660 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:12.607571    2563 generic.go:356] "Generic (PLEG): container finished" podID="5dc77547-44bd-4735-aef4-591c3160411a" containerID="59892782a41877919ff8ac68e08d27f27dc93c718df4ce657be59b704be08a22" exitCode=6
Jul 18 01:34:12.607660 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:12.607620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" event={"ID":"5dc77547-44bd-4735-aef4-591c3160411a","Type":"ContainerDied","Data":"59892782a41877919ff8ac68e08d27f27dc93c718df4ce657be59b704be08a22"}
Jul 18 01:34:12.608162 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:12.607675    2563 scope.go:117] "RemoveContainer" containerID="35e8401e15d2a5d22885d79469373775152a664d9407931968c564dccd763f08"
Jul 18 01:34:12.608162 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:12.607945    2563 scope.go:117] "RemoveContainer" containerID="59892782a41877919ff8ac68e08d27f27dc93c718df4ce657be59b704be08a22"
Jul 18 01:34:12.608260 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:12.608238    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj_odh-ai-gateway-infra(5dc77547-44bd-4735-aef4-591c3160411a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" podUID="5dc77547-44bd-4735-aef4-591c3160411a"
Jul 18 01:34:13.616375 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:13.616347    2563 scope.go:117] "RemoveContainer" containerID="59892782a41877919ff8ac68e08d27f27dc93c718df4ce657be59b704be08a22"
Jul 18 01:34:13.616763 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:13.616530    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj_odh-ai-gateway-infra(5dc77547-44bd-4735-aef4-591c3160411a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" podUID="5dc77547-44bd-4735-aef4-591c3160411a"
Jul 18 01:34:14.939466 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:14.939422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:34:24.170218 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:24.170123    2563 scope.go:117] "RemoveContainer" containerID="bc81e667bbe55703882e8060de83fc6a3b4aa96025d8dce6136c0841cd5d01ab"
Jul 18 01:34:24.685148 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:24.685105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" event={"ID":"cb8d14e4-5614-410f-becc-e66da26c9bfa","Type":"ContainerStarted","Data":"1f7bd595d2d90626f2a5823694b857a930de122b6477387b07106e3e31d2b41c"}
Jul 18 01:34:25.694257 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:25.694222    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerID="1f7bd595d2d90626f2a5823694b857a930de122b6477387b07106e3e31d2b41c" exitCode=6
Jul 18 01:34:25.694716 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:25.694268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" event={"ID":"cb8d14e4-5614-410f-becc-e66da26c9bfa","Type":"ContainerDied","Data":"1f7bd595d2d90626f2a5823694b857a930de122b6477387b07106e3e31d2b41c"}
Jul 18 01:34:25.694716 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:25.694316    2563 scope.go:117] "RemoveContainer" containerID="bc81e667bbe55703882e8060de83fc6a3b4aa96025d8dce6136c0841cd5d01ab"
Jul 18 01:34:25.694716 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:25.694564    2563 scope.go:117] "RemoveContainer" containerID="1f7bd595d2d90626f2a5823694b857a930de122b6477387b07106e3e31d2b41c"
Jul 18 01:34:25.694866 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:25.694846    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-qwtbx_odh-ai-gateway-infra(cb8d14e4-5614-410f-becc-e66da26c9bfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa"
Jul 18 01:34:25.709388 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:25.709358    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"]
Jul 18 01:34:26.839289 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:26.839262    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"
Jul 18 01:34:26.876149 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:26.876116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8d14e4-5614-410f-becc-e66da26c9bfa-maas-api-service-ca\") pod \"cb8d14e4-5614-410f-becc-e66da26c9bfa\" (UID: \"cb8d14e4-5614-410f-becc-e66da26c9bfa\") "
Jul 18 01:34:26.876505 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:26.876486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8d14e4-5614-410f-becc-e66da26c9bfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb8d14e4-5614-410f-becc-e66da26c9bfa" (UID: "cb8d14e4-5614-410f-becc-e66da26c9bfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:34:26.977269 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:26.977175    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8d14e4-5614-410f-becc-e66da26c9bfa-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:34:27.487572 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.487532    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"]
Jul 18 01:34:27.487949 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.487936    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:27.487990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.487950    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:27.487990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.487969    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:27.487990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.487974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:27.488130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.488038    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:27.488130 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.488048    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:27.491369 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.491351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"
Jul 18 01:34:27.497400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.497375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"]
Jul 18 01:34:27.583389 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.583342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df73ea-c1a0-4c3f-b775-e2a628f4252f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7\" (UID: \"01df73ea-c1a0-4c3f-b775-e2a628f4252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"
Jul 18 01:34:27.683917 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.683868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df73ea-c1a0-4c3f-b775-e2a628f4252f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7\" (UID: \"01df73ea-c1a0-4c3f-b775-e2a628f4252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"
Jul 18 01:34:27.684775 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.684748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df73ea-c1a0-4c3f-b775-e2a628f4252f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7\" (UID: \"01df73ea-c1a0-4c3f-b775-e2a628f4252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"
Jul 18 01:34:27.708693 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.708665    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"
Jul 18 01:34:27.708891 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.708694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx" event={"ID":"cb8d14e4-5614-410f-becc-e66da26c9bfa","Type":"ContainerDied","Data":"a7bb02dc434d7057805ba3d2474c8fa139e6cc2a6be87df753c4b9c8e0679bb3"}
Jul 18 01:34:27.708891 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.708766    2563 scope.go:117] "RemoveContainer" containerID="1f7bd595d2d90626f2a5823694b857a930de122b6477387b07106e3e31d2b41c"
Jul 18 01:34:27.732945 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.732914    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"]
Jul 18 01:34:27.740162 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.740102    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-qwtbx"]
Jul 18 01:34:27.802876 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.802825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"
Jul 18 01:34:27.942126 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:27.942096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"]
Jul 18 01:34:27.944366 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:34:27.944338    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01df73ea_c1a0_4c3f_b775_e2a628f4252f.slice/crio-96516772b6b3e4524eb53119f5f422d5dca007c18988cf564cb664512b3c4f7b WatchSource:0}: Error finding container 96516772b6b3e4524eb53119f5f422d5dca007c18988cf564cb664512b3c4f7b: Status 404 returned error can't find the container with id 96516772b6b3e4524eb53119f5f422d5dca007c18988cf564cb664512b3c4f7b
Jul 18 01:34:28.170548 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:28.170507    2563 scope.go:117] "RemoveContainer" containerID="59892782a41877919ff8ac68e08d27f27dc93c718df4ce657be59b704be08a22"
Jul 18 01:34:28.177385 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:28.177357    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" path="/var/lib/kubelet/pods/cb8d14e4-5614-410f-becc-e66da26c9bfa/volumes"
Jul 18 01:34:28.716888 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:28.716857    2563 generic.go:356] "Generic (PLEG): container finished" podID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerID="d8131b8e0eaf3d68888b1c2038c580b241c020865a62baa9d0005374ca1c75dc" exitCode=6
Jul 18 01:34:28.717097 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:28.716949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7" event={"ID":"01df73ea-c1a0-4c3f-b775-e2a628f4252f","Type":"ContainerDied","Data":"d8131b8e0eaf3d68888b1c2038c580b241c020865a62baa9d0005374ca1c75dc"}
Jul 18 01:34:28.717097 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:28.717012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7" event={"ID":"01df73ea-c1a0-4c3f-b775-e2a628f4252f","Type":"ContainerStarted","Data":"96516772b6b3e4524eb53119f5f422d5dca007c18988cf564cb664512b3c4f7b"}
Jul 18 01:34:28.717199 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:28.717167    2563 scope.go:117] "RemoveContainer" containerID="d8131b8e0eaf3d68888b1c2038c580b241c020865a62baa9d0005374ca1c75dc"
Jul 18 01:34:28.719998 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:28.719976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" event={"ID":"5dc77547-44bd-4735-aef4-591c3160411a","Type":"ContainerStarted","Data":"db35fafa672b314ce679cc7cfa38438096a7687b75a76311a5629082d72c90e0"}
Jul 18 01:34:29.727913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.727879    2563 generic.go:356] "Generic (PLEG): container finished" podID="5dc77547-44bd-4735-aef4-591c3160411a" containerID="db35fafa672b314ce679cc7cfa38438096a7687b75a76311a5629082d72c90e0" exitCode=6
Jul 18 01:34:29.728349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.727948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" event={"ID":"5dc77547-44bd-4735-aef4-591c3160411a","Type":"ContainerDied","Data":"db35fafa672b314ce679cc7cfa38438096a7687b75a76311a5629082d72c90e0"}
Jul 18 01:34:29.728349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.728013    2563 scope.go:117] "RemoveContainer" containerID="59892782a41877919ff8ac68e08d27f27dc93c718df4ce657be59b704be08a22"
Jul 18 01:34:29.728349 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.728306    2563 scope.go:117] "RemoveContainer" containerID="db35fafa672b314ce679cc7cfa38438096a7687b75a76311a5629082d72c90e0"
Jul 18 01:34:29.728539 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:29.728516    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj_odh-ai-gateway-infra(5dc77547-44bd-4735-aef4-591c3160411a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" podUID="5dc77547-44bd-4735-aef4-591c3160411a"
Jul 18 01:34:29.730249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.730227    2563 generic.go:356] "Generic (PLEG): container finished" podID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerID="ff9f6c6a99737d7067ebf379fac699e02a6b08f140a4889119ccf3d4c18d44f6" exitCode=6
Jul 18 01:34:29.730338 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.730295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7" event={"ID":"01df73ea-c1a0-4c3f-b775-e2a628f4252f","Type":"ContainerDied","Data":"ff9f6c6a99737d7067ebf379fac699e02a6b08f140a4889119ccf3d4c18d44f6"}
Jul 18 01:34:29.730487 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.730476    2563 scope.go:117] "RemoveContainer" containerID="ff9f6c6a99737d7067ebf379fac699e02a6b08f140a4889119ccf3d4c18d44f6"
Jul 18 01:34:29.730638 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:29.730624    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7_odh-ai-gateway-infra(01df73ea-c1a0-4c3f-b775-e2a628f4252f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f"
Jul 18 01:34:29.740140 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.740129    2563 scope.go:117] "RemoveContainer" containerID="d8131b8e0eaf3d68888b1c2038c580b241c020865a62baa9d0005374ca1c75dc"
Jul 18 01:34:29.753842 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:29.753819    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"]
Jul 18 01:34:30.740571 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:30.740546    2563 scope.go:117] "RemoveContainer" containerID="ff9f6c6a99737d7067ebf379fac699e02a6b08f140a4889119ccf3d4c18d44f6"
Jul 18 01:34:30.740896 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:30.740747    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7_odh-ai-gateway-infra(01df73ea-c1a0-4c3f-b775-e2a628f4252f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f"
Jul 18 01:34:30.886609 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:30.886579    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"
Jul 18 01:34:31.011426 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:31.011332    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc77547-44bd-4735-aef4-591c3160411a-maas-api-service-ca\") pod \"5dc77547-44bd-4735-aef4-591c3160411a\" (UID: \"5dc77547-44bd-4735-aef4-591c3160411a\") "
Jul 18 01:34:31.011696 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:31.011672    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc77547-44bd-4735-aef4-591c3160411a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5dc77547-44bd-4735-aef4-591c3160411a" (UID: "5dc77547-44bd-4735-aef4-591c3160411a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:34:31.112453 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:31.112421    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc77547-44bd-4735-aef4-591c3160411a-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:34:31.747200 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:31.747167    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"
Jul 18 01:34:31.747200 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:31.747190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj" event={"ID":"5dc77547-44bd-4735-aef4-591c3160411a","Type":"ContainerDied","Data":"16a288e58404f4e5bbed3c0cc1c6bf45999a4c6fb160a204ad007dbd0562dbc5"}
Jul 18 01:34:31.747683 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:31.747247    2563 scope.go:117] "RemoveContainer" containerID="db35fafa672b314ce679cc7cfa38438096a7687b75a76311a5629082d72c90e0"
Jul 18 01:34:31.774569 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:31.774536    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"]
Jul 18 01:34:31.778093 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:31.778050    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-fxvqj"]
Jul 18 01:34:32.176265 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.176178    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc77547-44bd-4735-aef4-591c3160411a" path="/var/lib/kubelet/pods/5dc77547-44bd-4735-aef4-591c3160411a/volumes"
Jul 18 01:34:32.561927 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.561889    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"]
Jul 18 01:34:32.562356 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562343    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:34:32.562419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562358    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:34:32.562419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562367    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:34:32.562419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562373    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:34:32.562419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562382    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:32.562419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562388    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:32.562419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562396    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:34:32.562419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562402    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:34:32.562640 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562462    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:34:32.562640 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562472    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8d14e4-5614-410f-becc-e66da26c9bfa" containerName="revoke-keys"
Jul 18 01:34:32.562640 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.562480    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:34:32.565746 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.565726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"
Jul 18 01:34:32.574626 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.574595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"]
Jul 18 01:34:32.625313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.625275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr\" (UID: \"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"
Jul 18 01:34:32.726563 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.726532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr\" (UID: \"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"
Jul 18 01:34:32.727186 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.727165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr\" (UID: \"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"
Jul 18 01:34:32.742266 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.742225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:34:32.878616 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:32.878532    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"
Jul 18 01:34:33.019581 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:33.019548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"]
Jul 18 01:34:33.020877 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:34:33.020854    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41b8b24_aa40_4dbb_ac1e_b1af9dcc95d1.slice/crio-bd3e9151b1be363f9be463c4671aa5786b506809d9a5bfaa530170acfd8f1721 WatchSource:0}: Error finding container bd3e9151b1be363f9be463c4671aa5786b506809d9a5bfaa530170acfd8f1721: Status 404 returned error can't find the container with id bd3e9151b1be363f9be463c4671aa5786b506809d9a5bfaa530170acfd8f1721
Jul 18 01:34:33.762295 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:33.762259    2563 generic.go:356] "Generic (PLEG): container finished" podID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerID="d517840816d7d6ddf71ae1f17bdce171d1de72513cf722396de3953fb7a8e8b2" exitCode=6
Jul 18 01:34:33.762474 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:33.762340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr" event={"ID":"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1","Type":"ContainerDied","Data":"d517840816d7d6ddf71ae1f17bdce171d1de72513cf722396de3953fb7a8e8b2"}
Jul 18 01:34:33.762474 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:33.762390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr" event={"ID":"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1","Type":"ContainerStarted","Data":"bd3e9151b1be363f9be463c4671aa5786b506809d9a5bfaa530170acfd8f1721"}
Jul 18 01:34:33.762555 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:33.762490    2563 scope.go:117] "RemoveContainer" containerID="d517840816d7d6ddf71ae1f17bdce171d1de72513cf722396de3953fb7a8e8b2"
Jul 18 01:34:34.769560 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:34.769527    2563 generic.go:356] "Generic (PLEG): container finished" podID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerID="e3cb590733aeff030bbdaf8c694b5a117d02b18dfbc150897c087a5a14e7dbe5" exitCode=6
Jul 18 01:34:34.769990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:34.769607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr" event={"ID":"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1","Type":"ContainerDied","Data":"e3cb590733aeff030bbdaf8c694b5a117d02b18dfbc150897c087a5a14e7dbe5"}
Jul 18 01:34:34.769990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:34.769673    2563 scope.go:117] "RemoveContainer" containerID="d517840816d7d6ddf71ae1f17bdce171d1de72513cf722396de3953fb7a8e8b2"
Jul 18 01:34:34.769990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:34.769899    2563 scope.go:117] "RemoveContainer" containerID="e3cb590733aeff030bbdaf8c694b5a117d02b18dfbc150897c087a5a14e7dbe5"
Jul 18 01:34:34.770157 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:34.770139    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr_odh-ai-gateway-infra(a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1"
Jul 18 01:34:35.777632 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:35.777602    2563 scope.go:117] "RemoveContainer" containerID="e3cb590733aeff030bbdaf8c694b5a117d02b18dfbc150897c087a5a14e7dbe5"
Jul 18 01:34:35.778028 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:35.777809    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr_odh-ai-gateway-infra(a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1"
Jul 18 01:34:42.169780 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:42.169745    2563 scope.go:117] "RemoveContainer" containerID="ff9f6c6a99737d7067ebf379fac699e02a6b08f140a4889119ccf3d4c18d44f6"
Jul 18 01:34:42.823954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:42.823922    2563 generic.go:356] "Generic (PLEG): container finished" podID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerID="f9b4b02cc4adb95b46836a2536f90b92799ad3c5340d13e2692936bfa03bec9c" exitCode=6
Jul 18 01:34:42.824138 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:42.823990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7" event={"ID":"01df73ea-c1a0-4c3f-b775-e2a628f4252f","Type":"ContainerDied","Data":"f9b4b02cc4adb95b46836a2536f90b92799ad3c5340d13e2692936bfa03bec9c"}
Jul 18 01:34:42.824138 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:42.824121    2563 scope.go:117] "RemoveContainer" containerID="ff9f6c6a99737d7067ebf379fac699e02a6b08f140a4889119ccf3d4c18d44f6"
Jul 18 01:34:42.824419 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:42.824403    2563 scope.go:117] "RemoveContainer" containerID="f9b4b02cc4adb95b46836a2536f90b92799ad3c5340d13e2692936bfa03bec9c"
Jul 18 01:34:42.824637 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:42.824617    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7_odh-ai-gateway-infra(01df73ea-c1a0-4c3f-b775-e2a628f4252f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f"
Jul 18 01:34:43.853498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:43.853459    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"]
Jul 18 01:34:43.992653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:43.992589    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"
Jul 18 01:34:44.136841 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:44.136762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df73ea-c1a0-4c3f-b775-e2a628f4252f-maas-api-service-ca\") pod \"01df73ea-c1a0-4c3f-b775-e2a628f4252f\" (UID: \"01df73ea-c1a0-4c3f-b775-e2a628f4252f\") "
Jul 18 01:34:44.137177 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:44.137148    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01df73ea-c1a0-4c3f-b775-e2a628f4252f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01df73ea-c1a0-4c3f-b775-e2a628f4252f" (UID: "01df73ea-c1a0-4c3f-b775-e2a628f4252f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:34:44.238415 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:44.238383    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01df73ea-c1a0-4c3f-b775-e2a628f4252f-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:34:44.838610 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:44.838577    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"
Jul 18 01:34:44.838787 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:44.838610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7" event={"ID":"01df73ea-c1a0-4c3f-b775-e2a628f4252f","Type":"ContainerDied","Data":"96516772b6b3e4524eb53119f5f422d5dca007c18988cf564cb664512b3c4f7b"}
Jul 18 01:34:44.838787 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:44.838673    2563 scope.go:117] "RemoveContainer" containerID="f9b4b02cc4adb95b46836a2536f90b92799ad3c5340d13e2692936bfa03bec9c"
Jul 18 01:34:44.855487 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:44.855456    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"]
Jul 18 01:34:44.863082 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:44.863025    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-lwcj7"]
Jul 18 01:34:46.177723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:46.177689    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" path="/var/lib/kubelet/pods/01df73ea-c1a0-4c3f-b775-e2a628f4252f/volumes"
Jul 18 01:34:51.169645 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:51.169612    2563 scope.go:117] "RemoveContainer" containerID="e3cb590733aeff030bbdaf8c694b5a117d02b18dfbc150897c087a5a14e7dbe5"
Jul 18 01:34:51.886546 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:51.886505    2563 generic.go:356] "Generic (PLEG): container finished" podID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerID="b3a47060ed035615aa5ccfecef10611465f4766c7ac08734cb4b5017620ed085" exitCode=6
Jul 18 01:34:51.886723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:51.886575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr" event={"ID":"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1","Type":"ContainerDied","Data":"b3a47060ed035615aa5ccfecef10611465f4766c7ac08734cb4b5017620ed085"}
Jul 18 01:34:51.886723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:51.886645    2563 scope.go:117] "RemoveContainer" containerID="e3cb590733aeff030bbdaf8c694b5a117d02b18dfbc150897c087a5a14e7dbe5"
Jul 18 01:34:51.887020 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:51.886998    2563 scope.go:117] "RemoveContainer" containerID="b3a47060ed035615aa5ccfecef10611465f4766c7ac08734cb4b5017620ed085"
Jul 18 01:34:51.887302 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:34:51.887276    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr_odh-ai-gateway-infra(a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1"
Jul 18 01:34:52.913343 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:52.913301    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"]
Jul 18 01:34:53.060853 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.060824    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"
Jul 18 01:34:53.215147 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.215116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1-maas-api-service-ca\") pod \"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1\" (UID: \"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1\") "
Jul 18 01:34:53.215524 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.215501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" (UID: "a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:34:53.316719 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.316681    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:34:53.902505 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.902479    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"
Jul 18 01:34:53.902505 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.902492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr" event={"ID":"a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1","Type":"ContainerDied","Data":"bd3e9151b1be363f9be463c4671aa5786b506809d9a5bfaa530170acfd8f1721"}
Jul 18 01:34:53.902725 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.902552    2563 scope.go:117] "RemoveContainer" containerID="b3a47060ed035615aa5ccfecef10611465f4766c7ac08734cb4b5017620ed085"
Jul 18 01:34:53.926029 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.925995    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"]
Jul 18 01:34:53.931498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:53.931469    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-5rslr"]
Jul 18 01:34:54.176920 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:54.176833    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" path="/var/lib/kubelet/pods/a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1/volumes"
Jul 18 01:34:54.742957 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:34:54.742917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:35:05.507790 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.507746    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"]
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508158    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508170    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508182    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508188    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508195    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508201    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508210    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508215    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508230    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508235    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:05.508292 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508296    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:05.508641 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508305    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc77547-44bd-4735-aef4-591c3160411a" containerName="revoke-keys"
Jul 18 01:35:05.508641 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508312    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:05.508641 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508319    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:05.508641 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.508325    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:05.512629 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.512610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"
Jul 18 01:35:05.515599 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.515573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:35:05.518665 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.518635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"]
Jul 18 01:35:05.525965 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.525931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99314a4-b730-4563-bf31-cb569a85d9b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-bd8j8\" (UID: \"e99314a4-b730-4563-bf31-cb569a85d9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"
Jul 18 01:35:05.627144 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.627097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99314a4-b730-4563-bf31-cb569a85d9b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-bd8j8\" (UID: \"e99314a4-b730-4563-bf31-cb569a85d9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"
Jul 18 01:35:05.627732 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.627709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99314a4-b730-4563-bf31-cb569a85d9b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-bd8j8\" (UID: \"e99314a4-b730-4563-bf31-cb569a85d9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"
Jul 18 01:35:05.825084 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.824985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"
Jul 18 01:35:05.986320 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:05.986285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"]
Jul 18 01:35:05.987008 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:35:05.986981    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99314a4_b730_4563_bf31_cb569a85d9b8.slice/crio-f04d47f3b1b47aedba9af8074694d15128990e5e70e7ecc1605fdeda3e58bf0a WatchSource:0}: Error finding container f04d47f3b1b47aedba9af8074694d15128990e5e70e7ecc1605fdeda3e58bf0a: Status 404 returned error can't find the container with id f04d47f3b1b47aedba9af8074694d15128990e5e70e7ecc1605fdeda3e58bf0a
Jul 18 01:35:06.984664 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:06.984626    2563 generic.go:356] "Generic (PLEG): container finished" podID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerID="3a0bbec201e6a65d6dbf96a49c82ff85b30ad8ee59982227c6d409fc0b033ee3" exitCode=6
Jul 18 01:35:06.985088 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:06.984716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8" event={"ID":"e99314a4-b730-4563-bf31-cb569a85d9b8","Type":"ContainerDied","Data":"3a0bbec201e6a65d6dbf96a49c82ff85b30ad8ee59982227c6d409fc0b033ee3"}
Jul 18 01:35:06.985088 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:06.984785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8" event={"ID":"e99314a4-b730-4563-bf31-cb569a85d9b8","Type":"ContainerStarted","Data":"f04d47f3b1b47aedba9af8074694d15128990e5e70e7ecc1605fdeda3e58bf0a"}
Jul 18 01:35:06.985088 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:06.984916    2563 scope.go:117] "RemoveContainer" containerID="3a0bbec201e6a65d6dbf96a49c82ff85b30ad8ee59982227c6d409fc0b033ee3"
Jul 18 01:35:07.992491 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:07.992453    2563 generic.go:356] "Generic (PLEG): container finished" podID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerID="884bb1d3e8c275a84bf09cc064517121e66fc9873939ccd2b25b6dd23432e7c0" exitCode=6
Jul 18 01:35:07.992893 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:07.992505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8" event={"ID":"e99314a4-b730-4563-bf31-cb569a85d9b8","Type":"ContainerDied","Data":"884bb1d3e8c275a84bf09cc064517121e66fc9873939ccd2b25b6dd23432e7c0"}
Jul 18 01:35:07.992893 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:07.992589    2563 scope.go:117] "RemoveContainer" containerID="3a0bbec201e6a65d6dbf96a49c82ff85b30ad8ee59982227c6d409fc0b033ee3"
Jul 18 01:35:07.992893 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:07.992838    2563 scope.go:117] "RemoveContainer" containerID="884bb1d3e8c275a84bf09cc064517121e66fc9873939ccd2b25b6dd23432e7c0"
Jul 18 01:35:07.993129 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:07.993102    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-bd8j8_odh-ai-gateway-infra(e99314a4-b730-4563-bf31-cb569a85d9b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8"
Jul 18 01:35:09.000147 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.000116    2563 scope.go:117] "RemoveContainer" containerID="884bb1d3e8c275a84bf09cc064517121e66fc9873939ccd2b25b6dd23432e7c0"
Jul 18 01:35:09.000554 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:09.000322    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-bd8j8_odh-ai-gateway-infra(e99314a4-b730-4563-bf31-cb569a85d9b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8"
Jul 18 01:35:09.454830 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.454795    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"]
Jul 18 01:35:09.455340 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.455322    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:09.455392 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.455343    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:09.455429 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.455417    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41b8b24-aa40-4dbb-ac1e-b1af9dcc95d1" containerName="revoke-keys"
Jul 18 01:35:09.455429 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.455426    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01df73ea-c1a0-4c3f-b775-e2a628f4252f" containerName="revoke-keys"
Jul 18 01:35:09.459909 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.459891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"
Jul 18 01:35:09.464440 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.464413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"]
Jul 18 01:35:09.557612 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.557568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d943e934-d1ba-41e5-84e3-3a1fa8e4ff42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-ffrss\" (UID: \"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"
Jul 18 01:35:09.658650 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.658604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d943e934-d1ba-41e5-84e3-3a1fa8e4ff42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-ffrss\" (UID: \"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"
Jul 18 01:35:09.659301 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.659275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d943e934-d1ba-41e5-84e3-3a1fa8e4ff42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-ffrss\" (UID: \"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"
Jul 18 01:35:09.772716 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:09.772608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"
Jul 18 01:35:10.116135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:10.116106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"]
Jul 18 01:35:10.117513 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:35:10.117483    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd943e934_d1ba_41e5_84e3_3a1fa8e4ff42.slice/crio-532627f57a2b02244954ec64776c22771ca4055531bbb6986d38d6fab150ff0b WatchSource:0}: Error finding container 532627f57a2b02244954ec64776c22771ca4055531bbb6986d38d6fab150ff0b: Status 404 returned error can't find the container with id 532627f57a2b02244954ec64776c22771ca4055531bbb6986d38d6fab150ff0b
Jul 18 01:35:11.015298 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:11.015261    2563 generic.go:356] "Generic (PLEG): container finished" podID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerID="5e275433df955a424b90288d364d48f0933523a16b8b4d120e4e256cd789e7b3" exitCode=6
Jul 18 01:35:11.015511 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:11.015343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss" event={"ID":"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42","Type":"ContainerDied","Data":"5e275433df955a424b90288d364d48f0933523a16b8b4d120e4e256cd789e7b3"}
Jul 18 01:35:11.015511 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:11.015396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss" event={"ID":"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42","Type":"ContainerStarted","Data":"532627f57a2b02244954ec64776c22771ca4055531bbb6986d38d6fab150ff0b"}
Jul 18 01:35:11.015633 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:11.015576    2563 scope.go:117] "RemoveContainer" containerID="5e275433df955a424b90288d364d48f0933523a16b8b4d120e4e256cd789e7b3"
Jul 18 01:35:12.029966 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:12.029927    2563 generic.go:356] "Generic (PLEG): container finished" podID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerID="84f244427281717b694d6c22acfdae306d50a1ddc0b36a0d1e57a770ce833ea6" exitCode=6
Jul 18 01:35:12.030465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:12.029996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss" event={"ID":"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42","Type":"ContainerDied","Data":"84f244427281717b694d6c22acfdae306d50a1ddc0b36a0d1e57a770ce833ea6"}
Jul 18 01:35:12.030465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:12.030104    2563 scope.go:117] "RemoveContainer" containerID="5e275433df955a424b90288d364d48f0933523a16b8b4d120e4e256cd789e7b3"
Jul 18 01:35:12.030465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:12.030296    2563 scope.go:117] "RemoveContainer" containerID="84f244427281717b694d6c22acfdae306d50a1ddc0b36a0d1e57a770ce833ea6"
Jul 18 01:35:12.030617 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:12.030552    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-ffrss_odh-ai-gateway-infra(d943e934-d1ba-41e5-84e3-3a1fa8e4ff42)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42"
Jul 18 01:35:13.037425 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:13.037397    2563 scope.go:117] "RemoveContainer" containerID="84f244427281717b694d6c22acfdae306d50a1ddc0b36a0d1e57a770ce833ea6"
Jul 18 01:35:13.037810 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:13.037572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-ffrss_odh-ai-gateway-infra(d943e934-d1ba-41e5-84e3-3a1fa8e4ff42)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42"
Jul 18 01:35:14.235079 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:14.235023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:35:17.530819 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:17.530781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"]
Jul 18 01:35:17.534493 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:17.534475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"
Jul 18 01:35:17.541131 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:17.541104    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"]
Jul 18 01:35:17.633080 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:17.633018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f34bbb5-9f55-4ba9-a4c0-d48c38390951-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687\" (UID: \"8f34bbb5-9f55-4ba9-a4c0-d48c38390951\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"
Jul 18 01:35:17.734537 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:17.734487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f34bbb5-9f55-4ba9-a4c0-d48c38390951-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687\" (UID: \"8f34bbb5-9f55-4ba9-a4c0-d48c38390951\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"
Jul 18 01:35:17.735173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:17.735151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f34bbb5-9f55-4ba9-a4c0-d48c38390951-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687\" (UID: \"8f34bbb5-9f55-4ba9-a4c0-d48c38390951\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"
Jul 18 01:35:17.847880 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:17.847790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"
Jul 18 01:35:17.992512 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:17.992484    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"]
Jul 18 01:35:17.995343 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:35:17.995317    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f34bbb5_9f55_4ba9_a4c0_d48c38390951.slice/crio-e21efc03cd5c8e440676ee6ad60222bfd1b2fa2ade9452ca8404f31496f014cd WatchSource:0}: Error finding container e21efc03cd5c8e440676ee6ad60222bfd1b2fa2ade9452ca8404f31496f014cd: Status 404 returned error can't find the container with id e21efc03cd5c8e440676ee6ad60222bfd1b2fa2ade9452ca8404f31496f014cd
Jul 18 01:35:18.068469 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:18.068432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687" event={"ID":"8f34bbb5-9f55-4ba9-a4c0-d48c38390951","Type":"ContainerStarted","Data":"e21efc03cd5c8e440676ee6ad60222bfd1b2fa2ade9452ca8404f31496f014cd"}
Jul 18 01:35:19.076035 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:19.075999    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerID="b97d4fcab9c159b0d137aaa9201699fd252a5b12aa6857075ba198f9e26d8843" exitCode=6
Jul 18 01:35:19.076442 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:19.076085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687" event={"ID":"8f34bbb5-9f55-4ba9-a4c0-d48c38390951","Type":"ContainerDied","Data":"b97d4fcab9c159b0d137aaa9201699fd252a5b12aa6857075ba198f9e26d8843"}
Jul 18 01:35:19.076442 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:19.076293    2563 scope.go:117] "RemoveContainer" containerID="b97d4fcab9c159b0d137aaa9201699fd252a5b12aa6857075ba198f9e26d8843"
Jul 18 01:35:20.084954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:20.084857    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerID="bba80a3090887f37e517124b1d8e493fc5668b56d60872a99f454c82f829dbda" exitCode=6
Jul 18 01:35:20.085446 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:20.084956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687" event={"ID":"8f34bbb5-9f55-4ba9-a4c0-d48c38390951","Type":"ContainerDied","Data":"bba80a3090887f37e517124b1d8e493fc5668b56d60872a99f454c82f829dbda"}
Jul 18 01:35:20.085446 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:20.085044    2563 scope.go:117] "RemoveContainer" containerID="b97d4fcab9c159b0d137aaa9201699fd252a5b12aa6857075ba198f9e26d8843"
Jul 18 01:35:20.085446 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:20.085208    2563 scope.go:117] "RemoveContainer" containerID="bba80a3090887f37e517124b1d8e493fc5668b56d60872a99f454c82f829dbda"
Jul 18 01:35:20.085615 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:20.085467    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687_odh-ai-gateway-infra(8f34bbb5-9f55-4ba9-a4c0-d48c38390951)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951"
Jul 18 01:35:21.093397 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:21.093363    2563 scope.go:117] "RemoveContainer" containerID="bba80a3090887f37e517124b1d8e493fc5668b56d60872a99f454c82f829dbda"
Jul 18 01:35:21.093856 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:21.093618    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687_odh-ai-gateway-infra(8f34bbb5-9f55-4ba9-a4c0-d48c38390951)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951"
Jul 18 01:35:23.170015 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:23.169976    2563 scope.go:117] "RemoveContainer" containerID="884bb1d3e8c275a84bf09cc064517121e66fc9873939ccd2b25b6dd23432e7c0"
Jul 18 01:35:24.113758 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:24.113667    2563 generic.go:356] "Generic (PLEG): container finished" podID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerID="8075e0ee5e478a0d73b01819b0925277d4bf52ed3c37107a9d3a8bfbeaf2276a" exitCode=6
Jul 18 01:35:24.113758 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:24.113739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8" event={"ID":"e99314a4-b730-4563-bf31-cb569a85d9b8","Type":"ContainerDied","Data":"8075e0ee5e478a0d73b01819b0925277d4bf52ed3c37107a9d3a8bfbeaf2276a"}
Jul 18 01:35:24.113963 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:24.113802    2563 scope.go:117] "RemoveContainer" containerID="884bb1d3e8c275a84bf09cc064517121e66fc9873939ccd2b25b6dd23432e7c0"
Jul 18 01:35:24.114121 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:24.114104    2563 scope.go:117] "RemoveContainer" containerID="8075e0ee5e478a0d73b01819b0925277d4bf52ed3c37107a9d3a8bfbeaf2276a"
Jul 18 01:35:24.114378 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:24.114346    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-bd8j8_odh-ai-gateway-infra(e99314a4-b730-4563-bf31-cb569a85d9b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8"
Jul 18 01:35:25.139827 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:25.139788    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"]
Jul 18 01:35:25.277900 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:25.277873    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"
Jul 18 01:35:25.404173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:25.404074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99314a4-b730-4563-bf31-cb569a85d9b8-maas-api-service-ca\") pod \"e99314a4-b730-4563-bf31-cb569a85d9b8\" (UID: \"e99314a4-b730-4563-bf31-cb569a85d9b8\") "
Jul 18 01:35:25.404467 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:25.404444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99314a4-b730-4563-bf31-cb569a85d9b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e99314a4-b730-4563-bf31-cb569a85d9b8" (UID: "e99314a4-b730-4563-bf31-cb569a85d9b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:35:25.505140 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:25.505102    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99314a4-b730-4563-bf31-cb569a85d9b8-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:35:26.128331 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:26.128296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"
Jul 18 01:35:26.128543 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:26.128352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8" event={"ID":"e99314a4-b730-4563-bf31-cb569a85d9b8","Type":"ContainerDied","Data":"f04d47f3b1b47aedba9af8074694d15128990e5e70e7ecc1605fdeda3e58bf0a"}
Jul 18 01:35:26.128543 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:26.128425    2563 scope.go:117] "RemoveContainer" containerID="8075e0ee5e478a0d73b01819b0925277d4bf52ed3c37107a9d3a8bfbeaf2276a"
Jul 18 01:35:26.154165 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:26.154134    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"]
Jul 18 01:35:26.160418 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:26.160392    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-bd8j8"]
Jul 18 01:35:26.175817 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:26.175787    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" path="/var/lib/kubelet/pods/e99314a4-b730-4563-bf31-cb569a85d9b8/volumes"
Jul 18 01:35:27.170048 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:27.170016    2563 scope.go:117] "RemoveContainer" containerID="84f244427281717b694d6c22acfdae306d50a1ddc0b36a0d1e57a770ce833ea6"
Jul 18 01:35:28.142880 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:28.142788    2563 generic.go:356] "Generic (PLEG): container finished" podID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerID="9daa831ae52cce011a53383cecbc61616a994bd7852df8318c2ec8d3be950abe" exitCode=6
Jul 18 01:35:28.142880 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:28.142837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss" event={"ID":"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42","Type":"ContainerDied","Data":"9daa831ae52cce011a53383cecbc61616a994bd7852df8318c2ec8d3be950abe"}
Jul 18 01:35:28.143118 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:28.142885    2563 scope.go:117] "RemoveContainer" containerID="84f244427281717b694d6c22acfdae306d50a1ddc0b36a0d1e57a770ce833ea6"
Jul 18 01:35:28.143228 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:28.143208    2563 scope.go:117] "RemoveContainer" containerID="9daa831ae52cce011a53383cecbc61616a994bd7852df8318c2ec8d3be950abe"
Jul 18 01:35:28.143460 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:28.143429    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-ffrss_odh-ai-gateway-infra(d943e934-d1ba-41e5-84e3-3a1fa8e4ff42)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42"
Jul 18 01:35:29.167541 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:29.167499    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"]
Jul 18 01:35:29.312528 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:29.312494    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"
Jul 18 01:35:29.441630 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:29.441553    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d943e934-d1ba-41e5-84e3-3a1fa8e4ff42-maas-api-service-ca\") pod \"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42\" (UID: \"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42\") "
Jul 18 01:35:29.442017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:29.441988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d943e934-d1ba-41e5-84e3-3a1fa8e4ff42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" (UID: "d943e934-d1ba-41e5-84e3-3a1fa8e4ff42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:35:29.542469 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:29.542419    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d943e934-d1ba-41e5-84e3-3a1fa8e4ff42-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:35:30.163469 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:30.163430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss" event={"ID":"d943e934-d1ba-41e5-84e3-3a1fa8e4ff42","Type":"ContainerDied","Data":"532627f57a2b02244954ec64776c22771ca4055531bbb6986d38d6fab150ff0b"}
Jul 18 01:35:30.163636 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:30.163515    2563 scope.go:117] "RemoveContainer" containerID="9daa831ae52cce011a53383cecbc61616a994bd7852df8318c2ec8d3be950abe"
Jul 18 01:35:30.163636 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:30.163448    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"
Jul 18 01:35:30.187124 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:30.187091    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"]
Jul 18 01:35:30.190722 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:30.190697    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-ffrss"]
Jul 18 01:35:32.176017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.175981    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" path="/var/lib/kubelet/pods/d943e934-d1ba-41e5-84e3-3a1fa8e4ff42/volumes"
Jul 18 01:35:32.607725 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.607683    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"]
Jul 18 01:35:32.608145 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608132    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:35:32.608199 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608148    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:35:32.608199 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608161    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:35:32.608199 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608167    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:35:32.608199 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608186    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:35:32.608199 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608193    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:35:32.608199 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608199    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:35:32.608400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608205    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:35:32.608400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608210    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:35:32.608400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608216    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:35:32.608400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608276    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:35:32.608400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608286    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:35:32.608400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608292    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:35:32.608400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608299    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:35:32.608400 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.608305    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:35:32.613003 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.612977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"
Jul 18 01:35:32.619179 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.619145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"]
Jul 18 01:35:32.636434 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.636382    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:35:32.672461 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.672419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88676c3e-4823-41d2-9e7f-2f8067be92c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v\" (UID: \"88676c3e-4823-41d2-9e7f-2f8067be92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"
Jul 18 01:35:32.773794 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.773757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88676c3e-4823-41d2-9e7f-2f8067be92c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v\" (UID: \"88676c3e-4823-41d2-9e7f-2f8067be92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"
Jul 18 01:35:32.774465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.774441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88676c3e-4823-41d2-9e7f-2f8067be92c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v\" (UID: \"88676c3e-4823-41d2-9e7f-2f8067be92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"
Jul 18 01:35:32.926241 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:32.926151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"
Jul 18 01:35:33.069885 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:33.069844    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"]
Jul 18 01:35:33.073259 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:35:33.073232    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88676c3e_4823_41d2_9e7f_2f8067be92c7.slice/crio-33a771e48bbcdbed3bac146ebe1f995b129f2dca5c8e281d14ea2e935ab0bfa9 WatchSource:0}: Error finding container 33a771e48bbcdbed3bac146ebe1f995b129f2dca5c8e281d14ea2e935ab0bfa9: Status 404 returned error can't find the container with id 33a771e48bbcdbed3bac146ebe1f995b129f2dca5c8e281d14ea2e935ab0bfa9
Jul 18 01:35:33.184961 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:33.184856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v" event={"ID":"88676c3e-4823-41d2-9e7f-2f8067be92c7","Type":"ContainerStarted","Data":"33a771e48bbcdbed3bac146ebe1f995b129f2dca5c8e281d14ea2e935ab0bfa9"}
Jul 18 01:35:34.192812 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:34.192714    2563 generic.go:356] "Generic (PLEG): container finished" podID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerID="05f668d9cc85823b3bbc624f4b399171c8733b39d2f7886d82c4d7de4834b674" exitCode=6
Jul 18 01:35:34.192812 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:34.192797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v" event={"ID":"88676c3e-4823-41d2-9e7f-2f8067be92c7","Type":"ContainerDied","Data":"05f668d9cc85823b3bbc624f4b399171c8733b39d2f7886d82c4d7de4834b674"}
Jul 18 01:35:34.193361 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:34.193011    2563 scope.go:117] "RemoveContainer" containerID="05f668d9cc85823b3bbc624f4b399171c8733b39d2f7886d82c4d7de4834b674"
Jul 18 01:35:35.201302 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:35.201268    2563 generic.go:356] "Generic (PLEG): container finished" podID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerID="ed67e79d3585bd9e27908f90adbf7152efdca9d35a07afde2e1aa333b3fb42e4" exitCode=6
Jul 18 01:35:35.201744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:35.201369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v" event={"ID":"88676c3e-4823-41d2-9e7f-2f8067be92c7","Type":"ContainerDied","Data":"ed67e79d3585bd9e27908f90adbf7152efdca9d35a07afde2e1aa333b3fb42e4"}
Jul 18 01:35:35.201744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:35.201426    2563 scope.go:117] "RemoveContainer" containerID="05f668d9cc85823b3bbc624f4b399171c8733b39d2f7886d82c4d7de4834b674"
Jul 18 01:35:35.201744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:35.201615    2563 scope.go:117] "RemoveContainer" containerID="ed67e79d3585bd9e27908f90adbf7152efdca9d35a07afde2e1aa333b3fb42e4"
Jul 18 01:35:35.201891 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:35.201869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v_odh-ai-gateway-infra(88676c3e-4823-41d2-9e7f-2f8067be92c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7"
Jul 18 01:35:36.169939 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:36.169907    2563 scope.go:117] "RemoveContainer" containerID="bba80a3090887f37e517124b1d8e493fc5668b56d60872a99f454c82f829dbda"
Jul 18 01:35:36.209115 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:36.209089    2563 scope.go:117] "RemoveContainer" containerID="ed67e79d3585bd9e27908f90adbf7152efdca9d35a07afde2e1aa333b3fb42e4"
Jul 18 01:35:36.209440 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:36.209291    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v_odh-ai-gateway-infra(88676c3e-4823-41d2-9e7f-2f8067be92c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7"
Jul 18 01:35:37.218294 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:37.218242    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerID="909768469f564982d63b330ebab3889e4c6f7196ef979b7862ba083a43bbe8ab" exitCode=6
Jul 18 01:35:37.218777 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:37.218309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687" event={"ID":"8f34bbb5-9f55-4ba9-a4c0-d48c38390951","Type":"ContainerDied","Data":"909768469f564982d63b330ebab3889e4c6f7196ef979b7862ba083a43bbe8ab"}
Jul 18 01:35:37.218777 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:37.218396    2563 scope.go:117] "RemoveContainer" containerID="bba80a3090887f37e517124b1d8e493fc5668b56d60872a99f454c82f829dbda"
Jul 18 01:35:37.218777 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:37.218735    2563 scope.go:117] "RemoveContainer" containerID="909768469f564982d63b330ebab3889e4c6f7196ef979b7862ba083a43bbe8ab"
Jul 18 01:35:37.219010 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:37.218974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687_odh-ai-gateway-infra(8f34bbb5-9f55-4ba9-a4c0-d48c38390951)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951"
Jul 18 01:35:38.245311 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:38.245277    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"]
Jul 18 01:35:38.392944 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:38.392911    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"
Jul 18 01:35:38.528030 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:38.527931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f34bbb5-9f55-4ba9-a4c0-d48c38390951-maas-api-service-ca\") pod \"8f34bbb5-9f55-4ba9-a4c0-d48c38390951\" (UID: \"8f34bbb5-9f55-4ba9-a4c0-d48c38390951\") "
Jul 18 01:35:38.528323 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:38.528299    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f34bbb5-9f55-4ba9-a4c0-d48c38390951-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f34bbb5-9f55-4ba9-a4c0-d48c38390951" (UID: "8f34bbb5-9f55-4ba9-a4c0-d48c38390951"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:35:38.629486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:38.629443    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f34bbb5-9f55-4ba9-a4c0-d48c38390951-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:35:39.234908 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:39.234875    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"
Jul 18 01:35:39.235113 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:39.234875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687" event={"ID":"8f34bbb5-9f55-4ba9-a4c0-d48c38390951","Type":"ContainerDied","Data":"e21efc03cd5c8e440676ee6ad60222bfd1b2fa2ade9452ca8404f31496f014cd"}
Jul 18 01:35:39.235113 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:39.235017    2563 scope.go:117] "RemoveContainer" containerID="909768469f564982d63b330ebab3889e4c6f7196ef979b7862ba083a43bbe8ab"
Jul 18 01:35:39.256846 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:39.256813    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"]
Jul 18 01:35:39.264303 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:39.264271    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-bk687"]
Jul 18 01:35:40.177002 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:40.176964    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" path="/var/lib/kubelet/pods/8f34bbb5-9f55-4ba9-a4c0-d48c38390951/volumes"
Jul 18 01:35:46.538743 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:46.538704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:35:51.170044 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:51.170004    2563 scope.go:117] "RemoveContainer" containerID="ed67e79d3585bd9e27908f90adbf7152efdca9d35a07afde2e1aa333b3fb42e4"
Jul 18 01:35:52.316513 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:52.316480    2563 generic.go:356] "Generic (PLEG): container finished" podID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerID="5b443970cf4fb4a408711408d83e5e3a8e8b1606d3280dd2cbad0567b3363604" exitCode=6
Jul 18 01:35:52.317017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:52.316557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v" event={"ID":"88676c3e-4823-41d2-9e7f-2f8067be92c7","Type":"ContainerDied","Data":"5b443970cf4fb4a408711408d83e5e3a8e8b1606d3280dd2cbad0567b3363604"}
Jul 18 01:35:52.317017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:52.316625    2563 scope.go:117] "RemoveContainer" containerID="ed67e79d3585bd9e27908f90adbf7152efdca9d35a07afde2e1aa333b3fb42e4"
Jul 18 01:35:52.317017 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:52.316955    2563 scope.go:117] "RemoveContainer" containerID="5b443970cf4fb4a408711408d83e5e3a8e8b1606d3280dd2cbad0567b3363604"
Jul 18 01:35:52.317267 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:35:52.317246    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v_odh-ai-gateway-infra(88676c3e-4823-41d2-9e7f-2f8067be92c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7"
Jul 18 01:35:53.343027 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:53.342990    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"]
Jul 18 01:35:53.488433 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:53.488404    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"
Jul 18 01:35:53.558479 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:53.558444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88676c3e-4823-41d2-9e7f-2f8067be92c7-maas-api-service-ca\") pod \"88676c3e-4823-41d2-9e7f-2f8067be92c7\" (UID: \"88676c3e-4823-41d2-9e7f-2f8067be92c7\") "
Jul 18 01:35:53.558777 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:53.558753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88676c3e-4823-41d2-9e7f-2f8067be92c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88676c3e-4823-41d2-9e7f-2f8067be92c7" (UID: "88676c3e-4823-41d2-9e7f-2f8067be92c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:35:53.659721 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:53.659628    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88676c3e-4823-41d2-9e7f-2f8067be92c7-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:35:54.332208 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:54.332180    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"
Jul 18 01:35:54.332383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:54.332176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v" event={"ID":"88676c3e-4823-41d2-9e7f-2f8067be92c7","Type":"ContainerDied","Data":"33a771e48bbcdbed3bac146ebe1f995b129f2dca5c8e281d14ea2e935ab0bfa9"}
Jul 18 01:35:54.332383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:54.332313    2563 scope.go:117] "RemoveContainer" containerID="5b443970cf4fb4a408711408d83e5e3a8e8b1606d3280dd2cbad0567b3363604"
Jul 18 01:35:54.347439 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:54.347410    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"]
Jul 18 01:35:54.355428 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:54.355397    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-9d76v"]
Jul 18 01:35:56.176154 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:35:56.176116    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" path="/var/lib/kubelet/pods/88676c3e-4823-41d2-9e7f-2f8067be92c7/volumes"
Jul 18 01:36:00.248219 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:00.248182    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:36:05.550189 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550150    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"]
Jul 18 01:36:05.550587 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550549    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:36:05.550587 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550560    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d943e934-d1ba-41e5-84e3-3a1fa8e4ff42" containerName="revoke-keys"
Jul 18 01:36:05.550587 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550570    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:05.550587 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550577    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:05.550587 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550586    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550592    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550610    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550615    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550622    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550627    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550639    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550644    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550697    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550705    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550712    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550718    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550723    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99314a4-b730-4563-bf31-cb569a85d9b8" containerName="revoke-keys"
Jul 18 01:36:05.550756 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.550728    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:05.555459 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.555437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"
Jul 18 01:36:05.557859 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.557829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:36:05.559878 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.559845    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"]
Jul 18 01:36:05.651496 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.651453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1966fcb1-123c-4f3a-82fa-743edf169804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-8cv7c\" (UID: \"1966fcb1-123c-4f3a-82fa-743edf169804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"
Jul 18 01:36:05.752595 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.752557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1966fcb1-123c-4f3a-82fa-743edf169804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-8cv7c\" (UID: \"1966fcb1-123c-4f3a-82fa-743edf169804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"
Jul 18 01:36:05.753255 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.753233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1966fcb1-123c-4f3a-82fa-743edf169804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-8cv7c\" (UID: \"1966fcb1-123c-4f3a-82fa-743edf169804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"
Jul 18 01:36:05.868269 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:05.868162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"
Jul 18 01:36:06.009740 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:06.009708    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"]
Jul 18 01:36:06.011177 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:06.011148    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1966fcb1_123c_4f3a_82fa_743edf169804.slice/crio-1fea0d735d9b6f5fd1ef8e69891cef4553ed79b94a629c68488c7a3899bbf6d8 WatchSource:0}: Error finding container 1fea0d735d9b6f5fd1ef8e69891cef4553ed79b94a629c68488c7a3899bbf6d8: Status 404 returned error can't find the container with id 1fea0d735d9b6f5fd1ef8e69891cef4553ed79b94a629c68488c7a3899bbf6d8
Jul 18 01:36:06.417467 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:06.417426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" event={"ID":"1966fcb1-123c-4f3a-82fa-743edf169804","Type":"ContainerStarted","Data":"f680425d64e3b5f4f20f8faea99c90f70e3fed74c7269d8bacb5b0162d608ed3"}
Jul 18 01:36:06.417681 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:06.417520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" event={"ID":"1966fcb1-123c-4f3a-82fa-743edf169804","Type":"ContainerStarted","Data":"1fea0d735d9b6f5fd1ef8e69891cef4553ed79b94a629c68488c7a3899bbf6d8"}
Jul 18 01:36:06.434275 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:06.434214    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" podStartSLOduration=1.434198954 podStartE2EDuration="1.434198954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:36:06.431490096 +0000 UTC m=+3336.751467480" watchObservedRunningTime="2026-07-18 01:36:06.434198954 +0000 UTC m=+3336.754176336"
Jul 18 01:36:07.424603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:07.424568    2563 generic.go:356] "Generic (PLEG): container finished" podID="1966fcb1-123c-4f3a-82fa-743edf169804" containerID="f680425d64e3b5f4f20f8faea99c90f70e3fed74c7269d8bacb5b0162d608ed3" exitCode=6
Jul 18 01:36:07.425135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:07.424651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" event={"ID":"1966fcb1-123c-4f3a-82fa-743edf169804","Type":"ContainerDied","Data":"f680425d64e3b5f4f20f8faea99c90f70e3fed74c7269d8bacb5b0162d608ed3"}
Jul 18 01:36:07.425135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:07.424884    2563 scope.go:117] "RemoveContainer" containerID="f680425d64e3b5f4f20f8faea99c90f70e3fed74c7269d8bacb5b0162d608ed3"
Jul 18 01:36:08.432884 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:08.432799    2563 generic.go:356] "Generic (PLEG): container finished" podID="1966fcb1-123c-4f3a-82fa-743edf169804" containerID="b675f870065f76c094494e1654a5723dfa4306d2f80e7ded541b1b0a94e0dc88" exitCode=6
Jul 18 01:36:08.432884 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:08.432864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" event={"ID":"1966fcb1-123c-4f3a-82fa-743edf169804","Type":"ContainerDied","Data":"b675f870065f76c094494e1654a5723dfa4306d2f80e7ded541b1b0a94e0dc88"}
Jul 18 01:36:08.433388 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:08.432935    2563 scope.go:117] "RemoveContainer" containerID="f680425d64e3b5f4f20f8faea99c90f70e3fed74c7269d8bacb5b0162d608ed3"
Jul 18 01:36:08.433388 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:08.433117    2563 scope.go:117] "RemoveContainer" containerID="b675f870065f76c094494e1654a5723dfa4306d2f80e7ded541b1b0a94e0dc88"
Jul 18 01:36:08.433388 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:08.433377    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-8cv7c_odh-ai-gateway-infra(1966fcb1-123c-4f3a-82fa-743edf169804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" podUID="1966fcb1-123c-4f3a-82fa-743edf169804"
Jul 18 01:36:09.441298 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.441263    2563 scope.go:117] "RemoveContainer" containerID="b675f870065f76c094494e1654a5723dfa4306d2f80e7ded541b1b0a94e0dc88"
Jul 18 01:36:09.441771 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:09.441504    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-8cv7c_odh-ai-gateway-infra(1966fcb1-123c-4f3a-82fa-743edf169804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" podUID="1966fcb1-123c-4f3a-82fa-743edf169804"
Jul 18 01:36:09.496989 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.496956    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"]
Jul 18 01:36:09.497392 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.497378    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:09.497442 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.497394    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88676c3e-4823-41d2-9e7f-2f8067be92c7" containerName="revoke-keys"
Jul 18 01:36:09.497478 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.497472    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f34bbb5-9f55-4ba9-a4c0-d48c38390951" containerName="revoke-keys"
Jul 18 01:36:09.501879 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.501857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"
Jul 18 01:36:09.508733 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.508703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"]
Jul 18 01:36:09.588497 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.588455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837e4e84-e6bb-4165-ac53-6093401e2dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-jglvf\" (UID: \"837e4e84-e6bb-4165-ac53-6093401e2dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"
Jul 18 01:36:09.689474 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.689433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837e4e84-e6bb-4165-ac53-6093401e2dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-jglvf\" (UID: \"837e4e84-e6bb-4165-ac53-6093401e2dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"
Jul 18 01:36:09.690291 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.690261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837e4e84-e6bb-4165-ac53-6093401e2dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd91509-jglvf\" (UID: \"837e4e84-e6bb-4165-ac53-6093401e2dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"
Jul 18 01:36:09.814143 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.814105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"
Jul 18 01:36:09.957909 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:09.957878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"]
Jul 18 01:36:09.958905 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:09.958877    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837e4e84_e6bb_4165_ac53_6093401e2dda.slice/crio-c1affaa4fe524ec4a8244e5699e597e783d553a4ffafea68fb4c516c9bdad53f WatchSource:0}: Error finding container c1affaa4fe524ec4a8244e5699e597e783d553a4ffafea68fb4c516c9bdad53f: Status 404 returned error can't find the container with id c1affaa4fe524ec4a8244e5699e597e783d553a4ffafea68fb4c516c9bdad53f
Jul 18 01:36:10.450194 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:10.450152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" event={"ID":"837e4e84-e6bb-4165-ac53-6093401e2dda","Type":"ContainerStarted","Data":"dccd548f8ddb37bfd5f3f03dff442e13b380262c4e8dcf682535e1afa4dd8657"}
Jul 18 01:36:10.450615 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:10.450214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" event={"ID":"837e4e84-e6bb-4165-ac53-6093401e2dda","Type":"ContainerStarted","Data":"c1affaa4fe524ec4a8244e5699e597e783d553a4ffafea68fb4c516c9bdad53f"}
Jul 18 01:36:10.468244 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:10.468186    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" podStartSLOduration=1.468169153 podStartE2EDuration="1.468169153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:36:10.466219586 +0000 UTC m=+3340.786196969" watchObservedRunningTime="2026-07-18 01:36:10.468169153 +0000 UTC m=+3340.788146536"
Jul 18 01:36:11.457438 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:11.457400    2563 generic.go:356] "Generic (PLEG): container finished" podID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerID="dccd548f8ddb37bfd5f3f03dff442e13b380262c4e8dcf682535e1afa4dd8657" exitCode=6
Jul 18 01:36:11.457855 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:11.457440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" event={"ID":"837e4e84-e6bb-4165-ac53-6093401e2dda","Type":"ContainerDied","Data":"dccd548f8ddb37bfd5f3f03dff442e13b380262c4e8dcf682535e1afa4dd8657"}
Jul 18 01:36:11.457855 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:11.457692    2563 scope.go:117] "RemoveContainer" containerID="dccd548f8ddb37bfd5f3f03dff442e13b380262c4e8dcf682535e1afa4dd8657"
Jul 18 01:36:12.465018 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:12.464979    2563 generic.go:356] "Generic (PLEG): container finished" podID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerID="89185970a720723b994a8b855d753348d14cc07bf74b110aacc1301e88afdca2" exitCode=6
Jul 18 01:36:12.465489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:12.465078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" event={"ID":"837e4e84-e6bb-4165-ac53-6093401e2dda","Type":"ContainerDied","Data":"89185970a720723b994a8b855d753348d14cc07bf74b110aacc1301e88afdca2"}
Jul 18 01:36:12.465489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:12.465147    2563 scope.go:117] "RemoveContainer" containerID="dccd548f8ddb37bfd5f3f03dff442e13b380262c4e8dcf682535e1afa4dd8657"
Jul 18 01:36:12.465489 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:12.465429    2563 scope.go:117] "RemoveContainer" containerID="89185970a720723b994a8b855d753348d14cc07bf74b110aacc1301e88afdca2"
Jul 18 01:36:12.465635 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:12.465615    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-jglvf_odh-ai-gateway-infra(837e4e84-e6bb-4165-ac53-6093401e2dda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda"
Jul 18 01:36:13.473619 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:13.473595    2563 scope.go:117] "RemoveContainer" containerID="89185970a720723b994a8b855d753348d14cc07bf74b110aacc1301e88afdca2"
Jul 18 01:36:13.474006 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:13.473802    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-jglvf_odh-ai-gateway-infra(837e4e84-e6bb-4165-ac53-6093401e2dda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda"
Jul 18 01:36:17.570392 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:17.570355    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"]
Jul 18 01:36:17.573784 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:17.573766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"
Jul 18 01:36:17.594104 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:17.594032    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"]
Jul 18 01:36:17.664843 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:17.664796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ace853-40f3-416b-bcb3-168cad95a7a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh\" (UID: \"78ace853-40f3-416b-bcb3-168cad95a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"
Jul 18 01:36:17.765498 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:17.765448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ace853-40f3-416b-bcb3-168cad95a7a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh\" (UID: \"78ace853-40f3-416b-bcb3-168cad95a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"
Jul 18 01:36:17.766207 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:17.766182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ace853-40f3-416b-bcb3-168cad95a7a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh\" (UID: \"78ace853-40f3-416b-bcb3-168cad95a7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"
Jul 18 01:36:17.891427 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:17.891317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"
Jul 18 01:36:18.036449 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:18.036408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"]
Jul 18 01:36:18.036722 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:18.036699    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ace853_40f3_416b_bcb3_168cad95a7a1.slice/crio-e04a6f5dde659c3b094638985d35e224a44c3f651c0d0ed763f4ca4462566525 WatchSource:0}: Error finding container e04a6f5dde659c3b094638985d35e224a44c3f651c0d0ed763f4ca4462566525: Status 404 returned error can't find the container with id e04a6f5dde659c3b094638985d35e224a44c3f651c0d0ed763f4ca4462566525
Jul 18 01:36:18.507187 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:18.507149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" event={"ID":"78ace853-40f3-416b-bcb3-168cad95a7a1","Type":"ContainerStarted","Data":"a88054a4a8b046c115785827a77468641390a1e084a62c1d266e61db01cb8d79"}
Jul 18 01:36:18.507465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:18.507205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" event={"ID":"78ace853-40f3-416b-bcb3-168cad95a7a1","Type":"ContainerStarted","Data":"e04a6f5dde659c3b094638985d35e224a44c3f651c0d0ed763f4ca4462566525"}
Jul 18 01:36:18.521993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:18.521940    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" podStartSLOduration=1.521923713 podStartE2EDuration="1.521923713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:36:18.521097679 +0000 UTC m=+3348.841075053" watchObservedRunningTime="2026-07-18 01:36:18.521923713 +0000 UTC m=+3348.841901146"
Jul 18 01:36:19.514606 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:19.514571    2563 generic.go:356] "Generic (PLEG): container finished" podID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerID="a88054a4a8b046c115785827a77468641390a1e084a62c1d266e61db01cb8d79" exitCode=6
Jul 18 01:36:19.515029 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:19.514658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" event={"ID":"78ace853-40f3-416b-bcb3-168cad95a7a1","Type":"ContainerDied","Data":"a88054a4a8b046c115785827a77468641390a1e084a62c1d266e61db01cb8d79"}
Jul 18 01:36:19.515029 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:19.514809    2563 scope.go:117] "RemoveContainer" containerID="a88054a4a8b046c115785827a77468641390a1e084a62c1d266e61db01cb8d79"
Jul 18 01:36:20.522408 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:20.522371    2563 generic.go:356] "Generic (PLEG): container finished" podID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerID="c69d2a68bbc716c09894346fd6df25c1cabe8ce1ae70c4125a442068df8d9aaa" exitCode=6
Jul 18 01:36:20.522840 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:20.522468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" event={"ID":"78ace853-40f3-416b-bcb3-168cad95a7a1","Type":"ContainerDied","Data":"c69d2a68bbc716c09894346fd6df25c1cabe8ce1ae70c4125a442068df8d9aaa"}
Jul 18 01:36:20.522840 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:20.522540    2563 scope.go:117] "RemoveContainer" containerID="a88054a4a8b046c115785827a77468641390a1e084a62c1d266e61db01cb8d79"
Jul 18 01:36:20.522840 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:20.522734    2563 scope.go:117] "RemoveContainer" containerID="c69d2a68bbc716c09894346fd6df25c1cabe8ce1ae70c4125a442068df8d9aaa"
Jul 18 01:36:20.522987 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:20.522969    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh_odh-ai-gateway-infra(78ace853-40f3-416b-bcb3-168cad95a7a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1"
Jul 18 01:36:21.344508 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:21.344474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:36:21.530551 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:21.530519    2563 scope.go:117] "RemoveContainer" containerID="c69d2a68bbc716c09894346fd6df25c1cabe8ce1ae70c4125a442068df8d9aaa"
Jul 18 01:36:21.530952 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:21.530705    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh_odh-ai-gateway-infra(78ace853-40f3-416b-bcb3-168cad95a7a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1"
Jul 18 01:36:23.169424 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:23.169391    2563 scope.go:117] "RemoveContainer" containerID="b675f870065f76c094494e1654a5723dfa4306d2f80e7ded541b1b0a94e0dc88"
Jul 18 01:36:23.545238 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:23.545190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" event={"ID":"1966fcb1-123c-4f3a-82fa-743edf169804","Type":"ContainerStarted","Data":"46d6bea685c88b79f79ff592682e0e9ccc57fd03446027da179798313605a4c2"}
Jul 18 01:36:24.553202 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:24.553163    2563 generic.go:356] "Generic (PLEG): container finished" podID="1966fcb1-123c-4f3a-82fa-743edf169804" containerID="46d6bea685c88b79f79ff592682e0e9ccc57fd03446027da179798313605a4c2" exitCode=6
Jul 18 01:36:24.553688 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:24.553233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" event={"ID":"1966fcb1-123c-4f3a-82fa-743edf169804","Type":"ContainerDied","Data":"46d6bea685c88b79f79ff592682e0e9ccc57fd03446027da179798313605a4c2"}
Jul 18 01:36:24.553688 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:24.553300    2563 scope.go:117] "RemoveContainer" containerID="b675f870065f76c094494e1654a5723dfa4306d2f80e7ded541b1b0a94e0dc88"
Jul 18 01:36:24.553688 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:24.553581    2563 scope.go:117] "RemoveContainer" containerID="46d6bea685c88b79f79ff592682e0e9ccc57fd03446027da179798313605a4c2"
Jul 18 01:36:24.553864 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:24.553842    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-8cv7c_odh-ai-gateway-infra(1966fcb1-123c-4f3a-82fa-743edf169804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" podUID="1966fcb1-123c-4f3a-82fa-743edf169804"
Jul 18 01:36:24.570260 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:24.570235    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"]
Jul 18 01:36:25.169513 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:25.169481    2563 scope.go:117] "RemoveContainer" containerID="89185970a720723b994a8b855d753348d14cc07bf74b110aacc1301e88afdca2"
Jul 18 01:36:25.563280 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:25.563249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" event={"ID":"837e4e84-e6bb-4165-ac53-6093401e2dda","Type":"ContainerStarted","Data":"12897165872dd9fd4e50081f9ff9327ef85d2fda0dcc312b6cd1019f9d2c2f74"}
Jul 18 01:36:25.694882 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:25.694856    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"
Jul 18 01:36:25.841501 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:25.841405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1966fcb1-123c-4f3a-82fa-743edf169804-maas-api-service-ca\") pod \"1966fcb1-123c-4f3a-82fa-743edf169804\" (UID: \"1966fcb1-123c-4f3a-82fa-743edf169804\") "
Jul 18 01:36:25.841744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:25.841717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1966fcb1-123c-4f3a-82fa-743edf169804-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1966fcb1-123c-4f3a-82fa-743edf169804" (UID: "1966fcb1-123c-4f3a-82fa-743edf169804"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:36:25.942421 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:25.942380    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1966fcb1-123c-4f3a-82fa-743edf169804-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:36:26.571580 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.571547    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"
Jul 18 01:36:26.571580 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.571555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c" event={"ID":"1966fcb1-123c-4f3a-82fa-743edf169804","Type":"ContainerDied","Data":"1fea0d735d9b6f5fd1ef8e69891cef4553ed79b94a629c68488c7a3899bbf6d8"}
Jul 18 01:36:26.572153 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.571622    2563 scope.go:117] "RemoveContainer" containerID="46d6bea685c88b79f79ff592682e0e9ccc57fd03446027da179798313605a4c2"
Jul 18 01:36:26.573842 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.573808    2563 generic.go:356] "Generic (PLEG): container finished" podID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerID="12897165872dd9fd4e50081f9ff9327ef85d2fda0dcc312b6cd1019f9d2c2f74" exitCode=6
Jul 18 01:36:26.573968 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.573851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" event={"ID":"837e4e84-e6bb-4165-ac53-6093401e2dda","Type":"ContainerDied","Data":"12897165872dd9fd4e50081f9ff9327ef85d2fda0dcc312b6cd1019f9d2c2f74"}
Jul 18 01:36:26.574266 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.574250    2563 scope.go:117] "RemoveContainer" containerID="12897165872dd9fd4e50081f9ff9327ef85d2fda0dcc312b6cd1019f9d2c2f74"
Jul 18 01:36:26.574463 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:26.574446    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9bd91509-jglvf_odh-ai-gateway-infra(837e4e84-e6bb-4165-ac53-6093401e2dda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda"
Jul 18 01:36:26.582295 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.582277    2563 scope.go:117] "RemoveContainer" containerID="89185970a720723b994a8b855d753348d14cc07bf74b110aacc1301e88afdca2"
Jul 18 01:36:26.602317 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.602260    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"]
Jul 18 01:36:26.612016 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.611982    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"]
Jul 18 01:36:26.614385 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:26.614358    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-8cv7c"]
Jul 18 01:36:27.728847 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:27.728820    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"
Jul 18 01:36:27.858417 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:27.858328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837e4e84-e6bb-4165-ac53-6093401e2dda-maas-api-service-ca\") pod \"837e4e84-e6bb-4165-ac53-6093401e2dda\" (UID: \"837e4e84-e6bb-4165-ac53-6093401e2dda\") "
Jul 18 01:36:27.858702 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:27.858675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837e4e84-e6bb-4165-ac53-6093401e2dda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "837e4e84-e6bb-4165-ac53-6093401e2dda" (UID: "837e4e84-e6bb-4165-ac53-6093401e2dda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:36:27.959789 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:27.959750    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/837e4e84-e6bb-4165-ac53-6093401e2dda-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:36:28.176166 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:28.176088    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" path="/var/lib/kubelet/pods/1966fcb1-123c-4f3a-82fa-743edf169804/volumes"
Jul 18 01:36:28.591614 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:28.591583    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"
Jul 18 01:36:28.591614 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:28.591605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf" event={"ID":"837e4e84-e6bb-4165-ac53-6093401e2dda","Type":"ContainerDied","Data":"c1affaa4fe524ec4a8244e5699e597e783d553a4ffafea68fb4c516c9bdad53f"}
Jul 18 01:36:28.591865 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:28.591689    2563 scope.go:117] "RemoveContainer" containerID="12897165872dd9fd4e50081f9ff9327ef85d2fda0dcc312b6cd1019f9d2c2f74"
Jul 18 01:36:28.612342 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:28.612312    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"]
Jul 18 01:36:28.617538 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:28.617515    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd91509-jglvf"]
Jul 18 01:36:30.177142 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:30.177106    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" path="/var/lib/kubelet/pods/837e4e84-e6bb-4165-ac53-6093401e2dda/volumes"
Jul 18 01:36:31.080460 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:31.080356    2563 scope.go:117] "RemoveContainer" containerID="51e385c10cc4f3930a03f9a38f7339b4a0885f0aaad3c97cd81636cab6c4ea42"
Jul 18 01:36:32.652312 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652275    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"]
Jul 18 01:36:32.652735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652676    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:32.652735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652689    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:32.652735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652697    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:32.652735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652703    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:32.652735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652710    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:32.652735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:32.652735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652739    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:32.652990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652744    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:32.652990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652819    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:32.652990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652827    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:32.652990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652833    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:32.652990 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.652841    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:32.657378 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.657357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"
Jul 18 01:36:32.663407 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.663377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"]
Jul 18 01:36:32.804432 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.804392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447d8409-868c-45d8-8a96-c0340bf2f619-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q\" (UID: \"447d8409-868c-45d8-8a96-c0340bf2f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"
Jul 18 01:36:32.905445 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.905322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447d8409-868c-45d8-8a96-c0340bf2f619-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q\" (UID: \"447d8409-868c-45d8-8a96-c0340bf2f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"
Jul 18 01:36:32.905954 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.905932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447d8409-868c-45d8-8a96-c0340bf2f619-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q\" (UID: \"447d8409-868c-45d8-8a96-c0340bf2f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"
Jul 18 01:36:32.969579 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:32.969543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"
Jul 18 01:36:33.107093 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:33.107046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"]
Jul 18 01:36:33.108474 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:33.108447    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447d8409_868c_45d8_8a96_c0340bf2f619.slice/crio-8f3eed37e8de3259b774ad1afa56cf568fb176324ce1c73bf3a28fde4ddd5dbb WatchSource:0}: Error finding container 8f3eed37e8de3259b774ad1afa56cf568fb176324ce1c73bf3a28fde4ddd5dbb: Status 404 returned error can't find the container with id 8f3eed37e8de3259b774ad1afa56cf568fb176324ce1c73bf3a28fde4ddd5dbb
Jul 18 01:36:33.624072 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:33.623969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" event={"ID":"447d8409-868c-45d8-8a96-c0340bf2f619","Type":"ContainerStarted","Data":"f7ce2fcf337ad2b6ba4bcc305fc7ed2b35a1db1649d89d7ca350b44f225d761c"}
Jul 18 01:36:33.624072 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:33.624030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" event={"ID":"447d8409-868c-45d8-8a96-c0340bf2f619","Type":"ContainerStarted","Data":"8f3eed37e8de3259b774ad1afa56cf568fb176324ce1c73bf3a28fde4ddd5dbb"}
Jul 18 01:36:33.639513 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:33.639453    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" podStartSLOduration=1.639437372 podStartE2EDuration="1.639437372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:36:33.637801076 +0000 UTC m=+3363.957778461" watchObservedRunningTime="2026-07-18 01:36:33.639437372 +0000 UTC m=+3363.959414751"
Jul 18 01:36:34.631380 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:34.631348    2563 generic.go:356] "Generic (PLEG): container finished" podID="447d8409-868c-45d8-8a96-c0340bf2f619" containerID="f7ce2fcf337ad2b6ba4bcc305fc7ed2b35a1db1649d89d7ca350b44f225d761c" exitCode=6
Jul 18 01:36:34.631764 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:34.631442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" event={"ID":"447d8409-868c-45d8-8a96-c0340bf2f619","Type":"ContainerDied","Data":"f7ce2fcf337ad2b6ba4bcc305fc7ed2b35a1db1649d89d7ca350b44f225d761c"}
Jul 18 01:36:34.631764 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:34.631624    2563 scope.go:117] "RemoveContainer" containerID="f7ce2fcf337ad2b6ba4bcc305fc7ed2b35a1db1649d89d7ca350b44f225d761c"
Jul 18 01:36:35.169670 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:35.169595    2563 scope.go:117] "RemoveContainer" containerID="c69d2a68bbc716c09894346fd6df25c1cabe8ce1ae70c4125a442068df8d9aaa"
Jul 18 01:36:35.639208 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:35.639174    2563 generic.go:356] "Generic (PLEG): container finished" podID="447d8409-868c-45d8-8a96-c0340bf2f619" containerID="1a15415ac302ed8b487af214fe084aa1d7894208cff3c074a0fc5be4e1155794" exitCode=6
Jul 18 01:36:35.639674 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:35.639266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" event={"ID":"447d8409-868c-45d8-8a96-c0340bf2f619","Type":"ContainerDied","Data":"1a15415ac302ed8b487af214fe084aa1d7894208cff3c074a0fc5be4e1155794"}
Jul 18 01:36:35.639674 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:35.639330    2563 scope.go:117] "RemoveContainer" containerID="f7ce2fcf337ad2b6ba4bcc305fc7ed2b35a1db1649d89d7ca350b44f225d761c"
Jul 18 01:36:35.639674 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:35.639583    2563 scope.go:117] "RemoveContainer" containerID="1a15415ac302ed8b487af214fe084aa1d7894208cff3c074a0fc5be4e1155794"
Jul 18 01:36:35.639861 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:35.639837    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q_odh-ai-gateway-infra(447d8409-868c-45d8-8a96-c0340bf2f619)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" podUID="447d8409-868c-45d8-8a96-c0340bf2f619"
Jul 18 01:36:35.641300 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:35.641273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" event={"ID":"78ace853-40f3-416b-bcb3-168cad95a7a1","Type":"ContainerStarted","Data":"cafe0b5e6a0e0898be7f9da92357bae97ea9d987743b9dbc7d893f8e73b84308"}
Jul 18 01:36:36.649392 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:36.649361    2563 scope.go:117] "RemoveContainer" containerID="1a15415ac302ed8b487af214fe084aa1d7894208cff3c074a0fc5be4e1155794"
Jul 18 01:36:36.649837 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:36.649596    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q_odh-ai-gateway-infra(447d8409-868c-45d8-8a96-c0340bf2f619)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" podUID="447d8409-868c-45d8-8a96-c0340bf2f619"
Jul 18 01:36:36.651128 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:36.651104    2563 generic.go:356] "Generic (PLEG): container finished" podID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerID="cafe0b5e6a0e0898be7f9da92357bae97ea9d987743b9dbc7d893f8e73b84308" exitCode=6
Jul 18 01:36:36.651245 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:36.651173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" event={"ID":"78ace853-40f3-416b-bcb3-168cad95a7a1","Type":"ContainerDied","Data":"cafe0b5e6a0e0898be7f9da92357bae97ea9d987743b9dbc7d893f8e73b84308"}
Jul 18 01:36:36.651245 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:36.651227    2563 scope.go:117] "RemoveContainer" containerID="c69d2a68bbc716c09894346fd6df25c1cabe8ce1ae70c4125a442068df8d9aaa"
Jul 18 01:36:36.651460 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:36.651442    2563 scope.go:117] "RemoveContainer" containerID="cafe0b5e6a0e0898be7f9da92357bae97ea9d987743b9dbc7d893f8e73b84308"
Jul 18 01:36:36.651642 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:36.651623    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh_odh-ai-gateway-infra(78ace853-40f3-416b-bcb3-168cad95a7a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1"
Jul 18 01:36:36.679233 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:36.679188    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"]
Jul 18 01:36:36.750995 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:36.746858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:36:37.806340 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:37.806306    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"
Jul 18 01:36:37.952478 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:37.952443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ace853-40f3-416b-bcb3-168cad95a7a1-maas-api-service-ca\") pod \"78ace853-40f3-416b-bcb3-168cad95a7a1\" (UID: \"78ace853-40f3-416b-bcb3-168cad95a7a1\") "
Jul 18 01:36:37.952821 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:37.952797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ace853-40f3-416b-bcb3-168cad95a7a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78ace853-40f3-416b-bcb3-168cad95a7a1" (UID: "78ace853-40f3-416b-bcb3-168cad95a7a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:36:38.053341 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:38.053303    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ace853-40f3-416b-bcb3-168cad95a7a1-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:36:38.675907 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:38.675873    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"
Jul 18 01:36:38.676176 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:38.675902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh" event={"ID":"78ace853-40f3-416b-bcb3-168cad95a7a1","Type":"ContainerDied","Data":"e04a6f5dde659c3b094638985d35e224a44c3f651c0d0ed763f4ca4462566525"}
Jul 18 01:36:38.676176 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:38.675965    2563 scope.go:117] "RemoveContainer" containerID="cafe0b5e6a0e0898be7f9da92357bae97ea9d987743b9dbc7d893f8e73b84308"
Jul 18 01:36:38.692235 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:38.692205    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"]
Jul 18 01:36:38.694645 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:38.694621    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-d88sh"]
Jul 18 01:36:40.175752 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:40.175717    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" path="/var/lib/kubelet/pods/78ace853-40f3-416b-bcb3-168cad95a7a1/volumes"
Jul 18 01:36:45.889180 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889140    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-140539942281776"]
Jul 18 01:36:45.889791 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889758    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:45.889791 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889781    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:45.889921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889796    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.889921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889804    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.889921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889817    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:45.889921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889826    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:45.889921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889842    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.889921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889851    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.889921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889869    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.889921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.890339 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.889988    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="837e4e84-e6bb-4165-ac53-6093401e2dda" containerName="revoke-keys"
Jul 18 01:36:45.890339 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.890001    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.890339 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.890011    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.890339 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.890021    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1966fcb1-123c-4f3a-82fa-743edf169804" containerName="revoke-keys"
Jul 18 01:36:45.890339 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.890031    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ace853-40f3-416b-bcb3-168cad95a7a1" containerName="revoke-keys"
Jul 18 01:36:45.893559 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.893530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:45.896166 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.896139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zkmc5\""
Jul 18 01:36:45.898476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:45.898452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942281776"]
Jul 18 01:36:46.020611 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:46.020568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprc2\" (UniqueName: \"kubernetes.io/projected/71e52c87-010b-4c07-9db4-e927e3cfac6a-kube-api-access-fprc2\") pod \"test-curl-7679-140539942281776\" (UID: \"71e52c87-010b-4c07-9db4-e927e3cfac6a\") " pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:46.121852 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:46.121817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprc2\" (UniqueName: \"kubernetes.io/projected/71e52c87-010b-4c07-9db4-e927e3cfac6a-kube-api-access-fprc2\") pod \"test-curl-7679-140539942281776\" (UID: \"71e52c87-010b-4c07-9db4-e927e3cfac6a\") " pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:46.130453 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:46.130414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprc2\" (UniqueName: \"kubernetes.io/projected/71e52c87-010b-4c07-9db4-e927e3cfac6a-kube-api-access-fprc2\") pod \"test-curl-7679-140539942281776\" (UID: \"71e52c87-010b-4c07-9db4-e927e3cfac6a\") " pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:46.204931 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:46.204894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:46.346585 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:46.346556    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942281776"]
Jul 18 01:36:46.348311 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:46.348284    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71e52c87_010b_4c07_9db4_e927e3cfac6a.slice/crio-5c5e155d86acc969293eeef90f7dada2c31d770707ca668560e42c37542a72cc WatchSource:0}: Error finding container 5c5e155d86acc969293eeef90f7dada2c31d770707ca668560e42c37542a72cc: Status 404 returned error can't find the container with id 5c5e155d86acc969293eeef90f7dada2c31d770707ca668560e42c37542a72cc
Jul 18 01:36:46.349601 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:46.349584    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 01:36:46.727452 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:46.727413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942281776" event={"ID":"71e52c87-010b-4c07-9db4-e927e3cfac6a","Type":"ContainerStarted","Data":"5c5e155d86acc969293eeef90f7dada2c31d770707ca668560e42c37542a72cc"}
Jul 18 01:36:47.735042 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:47.735003    2563 generic.go:356] "Generic (PLEG): container finished" podID="71e52c87-010b-4c07-9db4-e927e3cfac6a" containerID="c80d8e8dcb2bd807cc4b91b1b37422180318358d49f16340b5da8e03246872b3" exitCode=0
Jul 18 01:36:47.735458 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:47.735091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942281776" event={"ID":"71e52c87-010b-4c07-9db4-e927e3cfac6a","Type":"ContainerDied","Data":"c80d8e8dcb2bd807cc4b91b1b37422180318358d49f16340b5da8e03246872b3"}
Jul 18 01:36:48.889864 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:48.889828    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:48.925865 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:48.925835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-140539942281776_71e52c87-010b-4c07-9db4-e927e3cfac6a/test-curl-7679-140539942281776/0.log"
Jul 18 01:36:48.947341 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:48.947302    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-140539942281776"]
Jul 18 01:36:48.951348 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:48.951321    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-140539942281776"]
Jul 18 01:36:49.049680 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.049591    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fprc2\" (UniqueName: \"kubernetes.io/projected/71e52c87-010b-4c07-9db4-e927e3cfac6a-kube-api-access-fprc2\") pod \"71e52c87-010b-4c07-9db4-e927e3cfac6a\" (UID: \"71e52c87-010b-4c07-9db4-e927e3cfac6a\") "
Jul 18 01:36:49.051663 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.051632    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e52c87-010b-4c07-9db4-e927e3cfac6a-kube-api-access-fprc2" (OuterVolumeSpecName: "kube-api-access-fprc2") pod "71e52c87-010b-4c07-9db4-e927e3cfac6a" (UID: "71e52c87-010b-4c07-9db4-e927e3cfac6a"). InnerVolumeSpecName "kube-api-access-fprc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:36:49.067762 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.067740    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-140539942195504"]
Jul 18 01:36:49.068158 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.068145    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e52c87-010b-4c07-9db4-e927e3cfac6a" containerName="test-curl-7679-140539942281776"
Jul 18 01:36:49.068205 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.068160    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e52c87-010b-4c07-9db4-e927e3cfac6a" containerName="test-curl-7679-140539942281776"
Jul 18 01:36:49.068250 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.068238    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e52c87-010b-4c07-9db4-e927e3cfac6a" containerName="test-curl-7679-140539942281776"
Jul 18 01:36:49.071307 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.071291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942195504"
Jul 18 01:36:49.076958 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.076936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942195504"]
Jul 18 01:36:49.150664 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.150623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2nn\" (UniqueName: \"kubernetes.io/projected/b0a323b0-d53c-4595-9abd-82530b4e3870-kube-api-access-hk2nn\") pod \"test-curl-7679-140539942195504\" (UID: \"b0a323b0-d53c-4595-9abd-82530b4e3870\") " pod="opendatahub/test-curl-7679-140539942195504"
Jul 18 01:36:49.150846 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.150736    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fprc2\" (UniqueName: \"kubernetes.io/projected/71e52c87-010b-4c07-9db4-e927e3cfac6a-kube-api-access-fprc2\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:36:49.169922 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.169896    2563 scope.go:117] "RemoveContainer" containerID="1a15415ac302ed8b487af214fe084aa1d7894208cff3c074a0fc5be4e1155794"
Jul 18 01:36:49.251341 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.251300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2nn\" (UniqueName: \"kubernetes.io/projected/b0a323b0-d53c-4595-9abd-82530b4e3870-kube-api-access-hk2nn\") pod \"test-curl-7679-140539942195504\" (UID: \"b0a323b0-d53c-4595-9abd-82530b4e3870\") " pod="opendatahub/test-curl-7679-140539942195504"
Jul 18 01:36:49.258933 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.258901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2nn\" (UniqueName: \"kubernetes.io/projected/b0a323b0-d53c-4595-9abd-82530b4e3870-kube-api-access-hk2nn\") pod \"test-curl-7679-140539942195504\" (UID: \"b0a323b0-d53c-4595-9abd-82530b4e3870\") " pod="opendatahub/test-curl-7679-140539942195504"
Jul 18 01:36:49.381956 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.381859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942195504"
Jul 18 01:36:49.521669 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.521634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942195504"]
Jul 18 01:36:49.523024 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:49.522996    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0a323b0_d53c_4595_9abd_82530b4e3870.slice/crio-5158f2c88aaa0e66456f266fc1ac8b5b458cb99cbba5828b3d63dc7957d487b8 WatchSource:0}: Error finding container 5158f2c88aaa0e66456f266fc1ac8b5b458cb99cbba5828b3d63dc7957d487b8: Status 404 returned error can't find the container with id 5158f2c88aaa0e66456f266fc1ac8b5b458cb99cbba5828b3d63dc7957d487b8
Jul 18 01:36:49.748486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.748447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" event={"ID":"447d8409-868c-45d8-8a96-c0340bf2f619","Type":"ContainerStarted","Data":"c245547432f3b02e05aee6205333cbc06e9c9f40b65537307c2b54caededbd53"}
Jul 18 01:36:49.749725 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.749700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:49.749872 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.749736    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5e155d86acc969293eeef90f7dada2c31d770707ca668560e42c37542a72cc"
Jul 18 01:36:49.751020 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:49.750993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942195504" event={"ID":"b0a323b0-d53c-4595-9abd-82530b4e3870","Type":"ContainerStarted","Data":"5158f2c88aaa0e66456f266fc1ac8b5b458cb99cbba5828b3d63dc7957d487b8"}
Jul 18 01:36:49.765685 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:49.765649    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-140539942281776\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="71e52c87-010b-4c07-9db4-e927e3cfac6a" pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:50.174509 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:50.174465    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-140539942281776\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="71e52c87-010b-4c07-9db4-e927e3cfac6a" pod="opendatahub/test-curl-7679-140539942281776"
Jul 18 01:36:50.175907 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:50.175877    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e52c87-010b-4c07-9db4-e927e3cfac6a" path="/var/lib/kubelet/pods/71e52c87-010b-4c07-9db4-e927e3cfac6a/volumes"
Jul 18 01:36:50.759118 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:50.759080    2563 generic.go:356] "Generic (PLEG): container finished" podID="447d8409-868c-45d8-8a96-c0340bf2f619" containerID="c245547432f3b02e05aee6205333cbc06e9c9f40b65537307c2b54caededbd53" exitCode=6
Jul 18 01:36:50.759118 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:50.759099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" event={"ID":"447d8409-868c-45d8-8a96-c0340bf2f619","Type":"ContainerDied","Data":"c245547432f3b02e05aee6205333cbc06e9c9f40b65537307c2b54caededbd53"}
Jul 18 01:36:50.759387 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:50.759159    2563 scope.go:117] "RemoveContainer" containerID="1a15415ac302ed8b487af214fe084aa1d7894208cff3c074a0fc5be4e1155794"
Jul 18 01:36:50.759491 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:50.759471    2563 scope.go:117] "RemoveContainer" containerID="c245547432f3b02e05aee6205333cbc06e9c9f40b65537307c2b54caededbd53"
Jul 18 01:36:50.759782 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:36:50.759756    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q_odh-ai-gateway-infra(447d8409-868c-45d8-8a96-c0340bf2f619)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" podUID="447d8409-868c-45d8-8a96-c0340bf2f619"
Jul 18 01:36:50.760795 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:50.760774    2563 generic.go:356] "Generic (PLEG): container finished" podID="b0a323b0-d53c-4595-9abd-82530b4e3870" containerID="45e6f0ee7c659c78a38ab828c4f75e1ead8ecd70e4e27ba3122d1a7af0ad1015" exitCode=0
Jul 18 01:36:50.760904 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:50.760832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942195504" event={"ID":"b0a323b0-d53c-4595-9abd-82530b4e3870","Type":"ContainerDied","Data":"45e6f0ee7c659c78a38ab828c4f75e1ead8ecd70e4e27ba3122d1a7af0ad1015"}
Jul 18 01:36:50.774381 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:50.774355    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"]
Jul 18 01:36:51.435727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.435686    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:36:51.926711 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.926680    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"
Jul 18 01:36:51.950086 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.950038    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942195504"
Jul 18 01:36:51.969536 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.969471    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-140539942195504_b0a323b0-d53c-4595-9abd-82530b4e3870/test-curl-7679-140539942195504/0.log"
Jul 18 01:36:51.983921 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.983880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447d8409-868c-45d8-8a96-c0340bf2f619-maas-api-service-ca\") pod \"447d8409-868c-45d8-8a96-c0340bf2f619\" (UID: \"447d8409-868c-45d8-8a96-c0340bf2f619\") "
Jul 18 01:36:51.984156 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.983953    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk2nn\" (UniqueName: \"kubernetes.io/projected/b0a323b0-d53c-4595-9abd-82530b4e3870-kube-api-access-hk2nn\") pod \"b0a323b0-d53c-4595-9abd-82530b4e3870\" (UID: \"b0a323b0-d53c-4595-9abd-82530b4e3870\") "
Jul 18 01:36:51.984279 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.984254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447d8409-868c-45d8-8a96-c0340bf2f619-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "447d8409-868c-45d8-8a96-c0340bf2f619" (UID: "447d8409-868c-45d8-8a96-c0340bf2f619"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:36:51.986136 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.986098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a323b0-d53c-4595-9abd-82530b4e3870-kube-api-access-hk2nn" (OuterVolumeSpecName: "kube-api-access-hk2nn") pod "b0a323b0-d53c-4595-9abd-82530b4e3870" (UID: "b0a323b0-d53c-4595-9abd-82530b4e3870"). InnerVolumeSpecName "kube-api-access-hk2nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:36:51.994687 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.994658    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-140539942195504"]
Jul 18 01:36:51.997163 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:51.997137    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-140539942195504"]
Jul 18 01:36:52.085332 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.085248    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/447d8409-868c-45d8-8a96-c0340bf2f619-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:36:52.085332 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.085275    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk2nn\" (UniqueName: \"kubernetes.io/projected/b0a323b0-d53c-4595-9abd-82530b4e3870-kube-api-access-hk2nn\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:36:52.143036 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143002    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:52.143442 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143427    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:52.143486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:52.143486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143455    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:52.143486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143460    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:52.143486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143467    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a323b0-d53c-4595-9abd-82530b4e3870" containerName="test-curl-7679-140539942195504"
Jul 18 01:36:52.143486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143473    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a323b0-d53c-4595-9abd-82530b4e3870" containerName="test-curl-7679-140539942195504"
Jul 18 01:36:52.143635 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143489    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:52.143635 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143494    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:52.143635 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143558    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:52.143635 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143566    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a323b0-d53c-4595-9abd-82530b4e3870" containerName="test-curl-7679-140539942195504"
Jul 18 01:36:52.143635 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.143573    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:52.146677 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.146659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:52.152387 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.152361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:52.181475 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.181437    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a323b0-d53c-4595-9abd-82530b4e3870" path="/var/lib/kubelet/pods/b0a323b0-d53c-4595-9abd-82530b4e3870/volumes"
Jul 18 01:36:52.185822 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.185792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4d45\" (UniqueName: \"kubernetes.io/projected/3326e6c2-1038-4a45-bc4e-cfc182e24b90-kube-api-access-g4d45\") pod \"test-curl-7679-140539942319424\" (UID: \"3326e6c2-1038-4a45-bc4e-cfc182e24b90\") " pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:52.286833 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.286797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4d45\" (UniqueName: \"kubernetes.io/projected/3326e6c2-1038-4a45-bc4e-cfc182e24b90-kube-api-access-g4d45\") pod \"test-curl-7679-140539942319424\" (UID: \"3326e6c2-1038-4a45-bc4e-cfc182e24b90\") " pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:52.298298 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.298269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4d45\" (UniqueName: \"kubernetes.io/projected/3326e6c2-1038-4a45-bc4e-cfc182e24b90-kube-api-access-g4d45\") pod \"test-curl-7679-140539942319424\" (UID: \"3326e6c2-1038-4a45-bc4e-cfc182e24b90\") " pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:52.457971 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.457922    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:52.593914 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.593882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:52.594429 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:52.594404    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3326e6c2_1038_4a45_bc4e_cfc182e24b90.slice/crio-125711e8a4c13ce97325a5a78d25698dd6fd4a7b4b24b225443e42ee861f62bb WatchSource:0}: Error finding container 125711e8a4c13ce97325a5a78d25698dd6fd4a7b4b24b225443e42ee861f62bb: Status 404 returned error can't find the container with id 125711e8a4c13ce97325a5a78d25698dd6fd4a7b4b24b225443e42ee861f62bb
Jul 18 01:36:52.774703 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.774621    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"
Jul 18 01:36:52.774703 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.774645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q" event={"ID":"447d8409-868c-45d8-8a96-c0340bf2f619","Type":"ContainerDied","Data":"8f3eed37e8de3259b774ad1afa56cf568fb176324ce1c73bf3a28fde4ddd5dbb"}
Jul 18 01:36:52.774952 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.774716    2563 scope.go:117] "RemoveContainer" containerID="c245547432f3b02e05aee6205333cbc06e9c9f40b65537307c2b54caededbd53"
Jul 18 01:36:52.775983 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.775906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942319424" event={"ID":"3326e6c2-1038-4a45-bc4e-cfc182e24b90","Type":"ContainerStarted","Data":"125711e8a4c13ce97325a5a78d25698dd6fd4a7b4b24b225443e42ee861f62bb"}
Jul 18 01:36:52.777578 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.777557    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942195504"
Jul 18 01:36:52.785264 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.785036    2563 scope.go:117] "RemoveContainer" containerID="45e6f0ee7c659c78a38ab828c4f75e1ead8ecd70e4e27ba3122d1a7af0ad1015"
Jul 18 01:36:52.796663 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.796636    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"]
Jul 18 01:36:52.801662 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:52.801636    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-c278q"]
Jul 18 01:36:53.786940 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:53.786898    2563 generic.go:356] "Generic (PLEG): container finished" podID="3326e6c2-1038-4a45-bc4e-cfc182e24b90" containerID="e8d8a7a986cd22440dc91099c59afd77f281339115221a58c454cfb07795cea5" exitCode=0
Jul 18 01:36:53.787410 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:53.787004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942319424" event={"ID":"3326e6c2-1038-4a45-bc4e-cfc182e24b90","Type":"ContainerDied","Data":"e8d8a7a986cd22440dc91099c59afd77f281339115221a58c454cfb07795cea5"}
Jul 18 01:36:54.178156 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:54.178039    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" path="/var/lib/kubelet/pods/447d8409-868c-45d8-8a96-c0340bf2f619/volumes"
Jul 18 01:36:54.957262 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:54.957232    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:54.975653 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:54.975622    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-140539942319424_3326e6c2-1038-4a45-bc4e-cfc182e24b90/test-curl-7679-140539942319424/0.log"
Jul 18 01:36:54.995722 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:54.995679    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:54.998270 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:54.998242    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:55.021290 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.021256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4d45\" (UniqueName: \"kubernetes.io/projected/3326e6c2-1038-4a45-bc4e-cfc182e24b90-kube-api-access-g4d45\") pod \"3326e6c2-1038-4a45-bc4e-cfc182e24b90\" (UID: \"3326e6c2-1038-4a45-bc4e-cfc182e24b90\") "
Jul 18 01:36:55.023381 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.023349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3326e6c2-1038-4a45-bc4e-cfc182e24b90-kube-api-access-g4d45" (OuterVolumeSpecName: "kube-api-access-g4d45") pod "3326e6c2-1038-4a45-bc4e-cfc182e24b90" (UID: "3326e6c2-1038-4a45-bc4e-cfc182e24b90"). InnerVolumeSpecName "kube-api-access-g4d45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:36:55.107910 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.107823    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:55.108264 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.108249    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3326e6c2-1038-4a45-bc4e-cfc182e24b90" containerName="test-curl-7679-140539942319424"
Jul 18 01:36:55.108313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.108266    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3326e6c2-1038-4a45-bc4e-cfc182e24b90" containerName="test-curl-7679-140539942319424"
Jul 18 01:36:55.108348 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.108324    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="447d8409-868c-45d8-8a96-c0340bf2f619" containerName="revoke-keys"
Jul 18 01:36:55.108348 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.108336    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3326e6c2-1038-4a45-bc4e-cfc182e24b90" containerName="test-curl-7679-140539942319424"
Jul 18 01:36:55.111416 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.111400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:55.118157 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.118089    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:55.122261 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.122238    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4d45\" (UniqueName: \"kubernetes.io/projected/3326e6c2-1038-4a45-bc4e-cfc182e24b90-kube-api-access-g4d45\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:36:55.223386 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.223347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzkrt\" (UniqueName: \"kubernetes.io/projected/bcfc3261-63c1-4a18-b241-1e85beadb511-kube-api-access-vzkrt\") pod \"test-curl-7679-140539942319424\" (UID: \"bcfc3261-63c1-4a18-b241-1e85beadb511\") " pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:55.324684 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.324642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzkrt\" (UniqueName: \"kubernetes.io/projected/bcfc3261-63c1-4a18-b241-1e85beadb511-kube-api-access-vzkrt\") pod \"test-curl-7679-140539942319424\" (UID: \"bcfc3261-63c1-4a18-b241-1e85beadb511\") " pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:55.332903 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.332877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzkrt\" (UniqueName: \"kubernetes.io/projected/bcfc3261-63c1-4a18-b241-1e85beadb511-kube-api-access-vzkrt\") pod \"test-curl-7679-140539942319424\" (UID: \"bcfc3261-63c1-4a18-b241-1e85beadb511\") " pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:55.423318 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.423221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:55.562953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.562919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:55.564871 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:55.564840    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcfc3261_63c1_4a18_b241_1e85beadb511.slice/crio-5c8fe783864faaba46921aa2411bbae51318c23d6db31b18ece396e4a1361fd9 WatchSource:0}: Error finding container 5c8fe783864faaba46921aa2411bbae51318c23d6db31b18ece396e4a1361fd9: Status 404 returned error can't find the container with id 5c8fe783864faaba46921aa2411bbae51318c23d6db31b18ece396e4a1361fd9
Jul 18 01:36:55.802004 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.801974    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:55.802202 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.802015    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="125711e8a4c13ce97325a5a78d25698dd6fd4a7b4b24b225443e42ee861f62bb"
Jul 18 01:36:55.803327 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.803293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942319424" event={"ID":"bcfc3261-63c1-4a18-b241-1e85beadb511","Type":"ContainerStarted","Data":"5c8fe783864faaba46921aa2411bbae51318c23d6db31b18ece396e4a1361fd9"}
Jul 18 01:36:55.813811 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:55.813779    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7679-140539942319424" oldPodUID="3326e6c2-1038-4a45-bc4e-cfc182e24b90" podUID="bcfc3261-63c1-4a18-b241-1e85beadb511"
Jul 18 01:36:56.175672 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:56.175589    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3326e6c2-1038-4a45-bc4e-cfc182e24b90" path="/var/lib/kubelet/pods/3326e6c2-1038-4a45-bc4e-cfc182e24b90/volumes"
Jul 18 01:36:56.810637 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:56.810601    2563 generic.go:356] "Generic (PLEG): container finished" podID="bcfc3261-63c1-4a18-b241-1e85beadb511" containerID="0d5bad016d0071764e715ec25c1893a6e93805fa3c6fa3cac709ad0af6ab01f2" exitCode=0
Jul 18 01:36:56.810923 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:56.810687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942319424" event={"ID":"bcfc3261-63c1-4a18-b241-1e85beadb511","Type":"ContainerDied","Data":"0d5bad016d0071764e715ec25c1893a6e93805fa3c6fa3cac709ad0af6ab01f2"}
Jul 18 01:36:57.969721 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:57.969690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:57.990236 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:57.990201    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-140539942319424_bcfc3261-63c1-4a18-b241-1e85beadb511/test-curl-7679-140539942319424/0.log"
Jul 18 01:36:58.012041 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.011998    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:58.014432 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.014409    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-140539942319424"]
Jul 18 01:36:58.048472 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.048439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzkrt\" (UniqueName: \"kubernetes.io/projected/bcfc3261-63c1-4a18-b241-1e85beadb511-kube-api-access-vzkrt\") pod \"bcfc3261-63c1-4a18-b241-1e85beadb511\" (UID: \"bcfc3261-63c1-4a18-b241-1e85beadb511\") "
Jul 18 01:36:58.050613 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.050589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfc3261-63c1-4a18-b241-1e85beadb511-kube-api-access-vzkrt" (OuterVolumeSpecName: "kube-api-access-vzkrt") pod "bcfc3261-63c1-4a18-b241-1e85beadb511" (UID: "bcfc3261-63c1-4a18-b241-1e85beadb511"). InnerVolumeSpecName "kube-api-access-vzkrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:36:58.144171 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.144076    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-140539942320432"]
Jul 18 01:36:58.145252 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.145222    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfc3261-63c1-4a18-b241-1e85beadb511" containerName="test-curl-7679-140539942319424"
Jul 18 01:36:58.145423 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.145410    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfc3261-63c1-4a18-b241-1e85beadb511" containerName="test-curl-7679-140539942319424"
Jul 18 01:36:58.145735 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.145716    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcfc3261-63c1-4a18-b241-1e85beadb511" containerName="test-curl-7679-140539942319424"
Jul 18 01:36:58.149222 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.149197    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzkrt\" (UniqueName: \"kubernetes.io/projected/bcfc3261-63c1-4a18-b241-1e85beadb511-kube-api-access-vzkrt\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:36:58.149893 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.149872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942320432"
Jul 18 01:36:58.151862 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.151835    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942320432"]
Jul 18 01:36:58.176490 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.176456    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcfc3261-63c1-4a18-b241-1e85beadb511" path="/var/lib/kubelet/pods/bcfc3261-63c1-4a18-b241-1e85beadb511/volumes"
Jul 18 01:36:58.250422 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.250378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqknk\" (UniqueName: \"kubernetes.io/projected/0029faff-742f-4c89-83e6-2cca451eb19e-kube-api-access-jqknk\") pod \"test-curl-7679-140539942320432\" (UID: \"0029faff-742f-4c89-83e6-2cca451eb19e\") " pod="opendatahub/test-curl-7679-140539942320432"
Jul 18 01:36:58.351723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.351673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqknk\" (UniqueName: \"kubernetes.io/projected/0029faff-742f-4c89-83e6-2cca451eb19e-kube-api-access-jqknk\") pod \"test-curl-7679-140539942320432\" (UID: \"0029faff-742f-4c89-83e6-2cca451eb19e\") " pod="opendatahub/test-curl-7679-140539942320432"
Jul 18 01:36:58.359650 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.359615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqknk\" (UniqueName: \"kubernetes.io/projected/0029faff-742f-4c89-83e6-2cca451eb19e-kube-api-access-jqknk\") pod \"test-curl-7679-140539942320432\" (UID: \"0029faff-742f-4c89-83e6-2cca451eb19e\") " pod="opendatahub/test-curl-7679-140539942320432"
Jul 18 01:36:58.460012 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.459978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942320432"
Jul 18 01:36:58.597519 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.597491    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-140539942320432"]
Jul 18 01:36:58.599667 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:36:58.599641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0029faff_742f_4c89_83e6_2cca451eb19e.slice/crio-9dfb0d463edb70edfb3305c0f956d85dc9beb79388387b958f345ca6325bc825 WatchSource:0}: Error finding container 9dfb0d463edb70edfb3305c0f956d85dc9beb79388387b958f345ca6325bc825: Status 404 returned error can't find the container with id 9dfb0d463edb70edfb3305c0f956d85dc9beb79388387b958f345ca6325bc825
Jul 18 01:36:58.827367 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.827280    2563 scope.go:117] "RemoveContainer" containerID="0d5bad016d0071764e715ec25c1893a6e93805fa3c6fa3cac709ad0af6ab01f2"
Jul 18 01:36:58.827367 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.827281    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942319424"
Jul 18 01:36:58.828734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:58.828701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942320432" event={"ID":"0029faff-742f-4c89-83e6-2cca451eb19e","Type":"ContainerStarted","Data":"9dfb0d463edb70edfb3305c0f956d85dc9beb79388387b958f345ca6325bc825"}
Jul 18 01:36:59.837453 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:59.837411    2563 generic.go:356] "Generic (PLEG): container finished" podID="0029faff-742f-4c89-83e6-2cca451eb19e" containerID="a11db304cabbdb641ab1b6ba5b9f7bc46c24150cd5bfd3b280d60da96798a39d" exitCode=0
Jul 18 01:36:59.837867 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:36:59.837507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-140539942320432" event={"ID":"0029faff-742f-4c89-83e6-2cca451eb19e","Type":"ContainerDied","Data":"a11db304cabbdb641ab1b6ba5b9f7bc46c24150cd5bfd3b280d60da96798a39d"}
Jul 18 01:37:00.986469 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:00.986432    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942320432"
Jul 18 01:37:01.005694 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:01.005662    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7679-140539942320432_0029faff-742f-4c89-83e6-2cca451eb19e/test-curl-7679-140539942320432/0.log"
Jul 18 01:37:01.026795 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:01.026760    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-140539942320432"]
Jul 18 01:37:01.030507 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:01.030467    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-140539942320432"]
Jul 18 01:37:01.077366 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:01.077333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqknk\" (UniqueName: \"kubernetes.io/projected/0029faff-742f-4c89-83e6-2cca451eb19e-kube-api-access-jqknk\") pod \"0029faff-742f-4c89-83e6-2cca451eb19e\" (UID: \"0029faff-742f-4c89-83e6-2cca451eb19e\") "
Jul 18 01:37:01.079573 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:01.079530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0029faff-742f-4c89-83e6-2cca451eb19e-kube-api-access-jqknk" (OuterVolumeSpecName: "kube-api-access-jqknk") pod "0029faff-742f-4c89-83e6-2cca451eb19e" (UID: "0029faff-742f-4c89-83e6-2cca451eb19e"). InnerVolumeSpecName "kube-api-access-jqknk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:37:01.178404 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:01.178308    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqknk\" (UniqueName: \"kubernetes.io/projected/0029faff-742f-4c89-83e6-2cca451eb19e-kube-api-access-jqknk\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:01.850304 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:01.850273    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dfb0d463edb70edfb3305c0f956d85dc9beb79388387b958f345ca6325bc825"
Jul 18 01:37:01.850304 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:01.850302    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-140539942320432"
Jul 18 01:37:01.860448 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:01.860418    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-140539942320432\" is forbidden: User \"system:node:ip-10-0-143-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-142.ec2.internal' and this object" podUID="0029faff-742f-4c89-83e6-2cca451eb19e" pod="opendatahub/test-curl-7679-140539942320432"
Jul 18 01:37:02.175531 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:02.175451    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0029faff-742f-4c89-83e6-2cca451eb19e" path="/var/lib/kubelet/pods/0029faff-742f-4c89-83e6-2cca451eb19e/volumes"
Jul 18 01:37:04.986627 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:04.986583    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"]
Jul 18 01:37:04.987039 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:04.986989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0029faff-742f-4c89-83e6-2cca451eb19e" containerName="test-curl-7679-140539942320432"
Jul 18 01:37:04.987039 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:04.987000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0029faff-742f-4c89-83e6-2cca451eb19e" containerName="test-curl-7679-140539942320432"
Jul 18 01:37:04.987139 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:04.987088    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0029faff-742f-4c89-83e6-2cca451eb19e" containerName="test-curl-7679-140539942320432"
Jul 18 01:37:04.990309 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:04.990293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"
Jul 18 01:37:04.993013 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:04.992974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmz4m\""
Jul 18 01:37:04.996899 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:04.996867    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"]
Jul 18 01:37:05.113980 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.113937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b3343d-1504-49fe-958c-74001f3e8d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h\" (UID: \"94b3343d-1504-49fe-958c-74001f3e8d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"
Jul 18 01:37:05.214650 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.214610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b3343d-1504-49fe-958c-74001f3e8d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h\" (UID: \"94b3343d-1504-49fe-958c-74001f3e8d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"
Jul 18 01:37:05.215311 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.215290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b3343d-1504-49fe-958c-74001f3e8d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h\" (UID: \"94b3343d-1504-49fe-958c-74001f3e8d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"
Jul 18 01:37:05.303223 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.303129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"
Jul 18 01:37:05.446188 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.446150    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"]
Jul 18 01:37:05.446992 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:37:05.446964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b3343d_1504_49fe_958c_74001f3e8d8e.slice/crio-b51c953cd87192f4ef32f89be88f1e5df1215f7c99b9b83d24d3da182bd6063f WatchSource:0}: Error finding container b51c953cd87192f4ef32f89be88f1e5df1215f7c99b9b83d24d3da182bd6063f: Status 404 returned error can't find the container with id b51c953cd87192f4ef32f89be88f1e5df1215f7c99b9b83d24d3da182bd6063f
Jul 18 01:37:05.593446 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.593353    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"]
Jul 18 01:37:05.598305 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.598277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"
Jul 18 01:37:05.603476 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.603442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"]
Jul 18 01:37:05.718937 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.718898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d98aa27-4cfe-4d34-9202-deb5774c2533-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-t6z65\" (UID: \"3d98aa27-4cfe-4d34-9202-deb5774c2533\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"
Jul 18 01:37:05.819456 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.819424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d98aa27-4cfe-4d34-9202-deb5774c2533-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-t6z65\" (UID: \"3d98aa27-4cfe-4d34-9202-deb5774c2533\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"
Jul 18 01:37:05.820144 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.820121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d98aa27-4cfe-4d34-9202-deb5774c2533-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4701d191-t6z65\" (UID: \"3d98aa27-4cfe-4d34-9202-deb5774c2533\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"
Jul 18 01:37:05.874326 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.874216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h" event={"ID":"94b3343d-1504-49fe-958c-74001f3e8d8e","Type":"ContainerStarted","Data":"e2b28be459b8f785ed6efd99fbc7b3bff352556f09c94c5d196c83ee276713ff"}
Jul 18 01:37:05.874326 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.874278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h" event={"ID":"94b3343d-1504-49fe-958c-74001f3e8d8e","Type":"ContainerStarted","Data":"b51c953cd87192f4ef32f89be88f1e5df1215f7c99b9b83d24d3da182bd6063f"}
Jul 18 01:37:05.892556 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.892486    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h" podStartSLOduration=1.892464426 podStartE2EDuration="1.892464426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:37:05.890128935 +0000 UTC m=+3396.210106318" watchObservedRunningTime="2026-07-18 01:37:05.892464426 +0000 UTC m=+3396.212441812"
Jul 18 01:37:05.911363 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:05.911326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"
Jul 18 01:37:06.056592 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:06.056559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"]
Jul 18 01:37:06.057306 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:37:06.057281    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d98aa27_4cfe_4d34_9202_deb5774c2533.slice/crio-b3fa99c9d1f31b3fdf0918aa747f63740eee0397aa61b01eabfab6959adb72cf WatchSource:0}: Error finding container b3fa99c9d1f31b3fdf0918aa747f63740eee0397aa61b01eabfab6959adb72cf: Status 404 returned error can't find the container with id b3fa99c9d1f31b3fdf0918aa747f63740eee0397aa61b01eabfab6959adb72cf
Jul 18 01:37:06.882387 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:06.882351    2563 generic.go:356] "Generic (PLEG): container finished" podID="94b3343d-1504-49fe-958c-74001f3e8d8e" containerID="e2b28be459b8f785ed6efd99fbc7b3bff352556f09c94c5d196c83ee276713ff" exitCode=0
Jul 18 01:37:06.882607 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:06.882428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h" event={"ID":"94b3343d-1504-49fe-958c-74001f3e8d8e","Type":"ContainerDied","Data":"e2b28be459b8f785ed6efd99fbc7b3bff352556f09c94c5d196c83ee276713ff"}
Jul 18 01:37:06.884034 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:06.884011    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerID="26610f45d4178f19500265a36313b400b5715c93f101d535d086a51ee9ee57d0" exitCode=6
Jul 18 01:37:06.884152 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:06.884094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65" event={"ID":"3d98aa27-4cfe-4d34-9202-deb5774c2533","Type":"ContainerDied","Data":"26610f45d4178f19500265a36313b400b5715c93f101d535d086a51ee9ee57d0"}
Jul 18 01:37:06.884152 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:06.884142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65" event={"ID":"3d98aa27-4cfe-4d34-9202-deb5774c2533","Type":"ContainerStarted","Data":"b3fa99c9d1f31b3fdf0918aa747f63740eee0397aa61b01eabfab6959adb72cf"}
Jul 18 01:37:06.884318 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:06.884306    2563 scope.go:117] "RemoveContainer" containerID="26610f45d4178f19500265a36313b400b5715c93f101d535d086a51ee9ee57d0"
Jul 18 01:37:07.892364 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:07.892271    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerID="49a6b5b97e65985ae9e58c1619f18ffb28800ebafeb8f3903e3992eec7da647c" exitCode=6
Jul 18 01:37:07.892808 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:07.892360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65" event={"ID":"3d98aa27-4cfe-4d34-9202-deb5774c2533","Type":"ContainerDied","Data":"49a6b5b97e65985ae9e58c1619f18ffb28800ebafeb8f3903e3992eec7da647c"}
Jul 18 01:37:07.892808 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:07.892457    2563 scope.go:117] "RemoveContainer" containerID="26610f45d4178f19500265a36313b400b5715c93f101d535d086a51ee9ee57d0"
Jul 18 01:37:07.892808 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:07.892566    2563 scope.go:117] "RemoveContainer" containerID="49a6b5b97e65985ae9e58c1619f18ffb28800ebafeb8f3903e3992eec7da647c"
Jul 18 01:37:07.892973 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:07.892808    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-t6z65_odh-ai-gateway-infra(3d98aa27-4cfe-4d34-9202-deb5774c2533)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533"
Jul 18 01:37:08.041319 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:08.041292    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"
Jul 18 01:37:08.144416 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:08.144312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b3343d-1504-49fe-958c-74001f3e8d8e-maas-api-service-ca\") pod \"94b3343d-1504-49fe-958c-74001f3e8d8e\" (UID: \"94b3343d-1504-49fe-958c-74001f3e8d8e\") "
Jul 18 01:37:08.144708 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:08.144682    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b3343d-1504-49fe-958c-74001f3e8d8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94b3343d-1504-49fe-958c-74001f3e8d8e" (UID: "94b3343d-1504-49fe-958c-74001f3e8d8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:37:08.245583 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:08.245542    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94b3343d-1504-49fe-958c-74001f3e8d8e-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:08.900437 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:08.900410    2563 scope.go:117] "RemoveContainer" containerID="49a6b5b97e65985ae9e58c1619f18ffb28800ebafeb8f3903e3992eec7da647c"
Jul 18 01:37:08.900928 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:08.900628    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-t6z65_odh-ai-gateway-infra(3d98aa27-4cfe-4d34-9202-deb5774c2533)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533"
Jul 18 01:37:08.901507 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:08.901475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h" event={"ID":"94b3343d-1504-49fe-958c-74001f3e8d8e","Type":"ContainerDied","Data":"b51c953cd87192f4ef32f89be88f1e5df1215f7c99b9b83d24d3da182bd6063f"}
Jul 18 01:37:08.901648 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:08.901505    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h"
Jul 18 01:37:08.901648 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:08.901526    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b51c953cd87192f4ef32f89be88f1e5df1215f7c99b9b83d24d3da182bd6063f"
Jul 18 01:37:15.397282 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.397244    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"]
Jul 18 01:37:15.397854 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.397505    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6" podUID="77620092-1cbc-486a-a296-5a7d54ec1bfa" containerName="maas-api" containerID="cri-o://4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a" gracePeriod=30
Jul 18 01:37:15.650759 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.650697    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:37:15.814981 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.814952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls\") pod \"77620092-1cbc-486a-a296-5a7d54ec1bfa\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") "
Jul 18 01:37:15.815198 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.815027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77620092-1cbc-486a-a296-5a7d54ec1bfa-openshift-service-ca\") pod \"77620092-1cbc-486a-a296-5a7d54ec1bfa\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") "
Jul 18 01:37:15.815198 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.815050    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjhlb\" (UniqueName: \"kubernetes.io/projected/77620092-1cbc-486a-a296-5a7d54ec1bfa-kube-api-access-cjhlb\") pod \"77620092-1cbc-486a-a296-5a7d54ec1bfa\" (UID: \"77620092-1cbc-486a-a296-5a7d54ec1bfa\") "
Jul 18 01:37:15.815445 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.815415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77620092-1cbc-486a-a296-5a7d54ec1bfa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77620092-1cbc-486a-a296-5a7d54ec1bfa" (UID: "77620092-1cbc-486a-a296-5a7d54ec1bfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:37:15.817217 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.817193    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77620092-1cbc-486a-a296-5a7d54ec1bfa" (UID: "77620092-1cbc-486a-a296-5a7d54ec1bfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:37:15.817310 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.817282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77620092-1cbc-486a-a296-5a7d54ec1bfa-kube-api-access-cjhlb" (OuterVolumeSpecName: "kube-api-access-cjhlb") pod "77620092-1cbc-486a-a296-5a7d54ec1bfa" (UID: "77620092-1cbc-486a-a296-5a7d54ec1bfa"). InnerVolumeSpecName "kube-api-access-cjhlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:37:15.916173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.916046    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77620092-1cbc-486a-a296-5a7d54ec1bfa-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:15.916173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.916118    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77620092-1cbc-486a-a296-5a7d54ec1bfa-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:15.916173 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.916128    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhlb\" (UniqueName: \"kubernetes.io/projected/77620092-1cbc-486a-a296-5a7d54ec1bfa-kube-api-access-cjhlb\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:15.947433 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.947400    2563 generic.go:356] "Generic (PLEG): container finished" podID="77620092-1cbc-486a-a296-5a7d54ec1bfa" containerID="4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a" exitCode=0
Jul 18 01:37:15.947649 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.947460    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"
Jul 18 01:37:15.947649 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.947489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6" event={"ID":"77620092-1cbc-486a-a296-5a7d54ec1bfa","Type":"ContainerDied","Data":"4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a"}
Jul 18 01:37:15.947649 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.947539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6" event={"ID":"77620092-1cbc-486a-a296-5a7d54ec1bfa","Type":"ContainerDied","Data":"44ec0f0896bacb22d892ea06aea32afbc6c549e9cfa447d98378cf5da4d7759b"}
Jul 18 01:37:15.947649 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.947560    2563 scope.go:117] "RemoveContainer" containerID="4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a"
Jul 18 01:37:15.957533 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.957510    2563 scope.go:117] "RemoveContainer" containerID="4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a"
Jul 18 01:37:15.957775 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:15.957757    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a\": container with ID starting with 4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a not found: ID does not exist" containerID="4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a"
Jul 18 01:37:15.957826 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.957781    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a"} err="failed to get container status \"4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a\": rpc error: code = NotFound desc = could not find container \"4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a\": container with ID starting with 4c07b055150abbce323ae5a52cb9e09f2c35a3841f7daa55544efb41c36b6e2a not found: ID does not exist"
Jul 18 01:37:15.968716 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.968684    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"]
Jul 18 01:37:15.972383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:15.972358    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ad1b67-57b8c4d855-78gg6"]
Jul 18 01:37:16.176450 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:16.176368    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77620092-1cbc-486a-a296-5a7d54ec1bfa" path="/var/lib/kubelet/pods/77620092-1cbc-486a-a296-5a7d54ec1bfa/volumes"
Jul 18 01:37:17.617333 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.617294    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"]
Jul 18 01:37:17.617734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.617689    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77620092-1cbc-486a-a296-5a7d54ec1bfa" containerName="maas-api"
Jul 18 01:37:17.617734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.617700    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="77620092-1cbc-486a-a296-5a7d54ec1bfa" containerName="maas-api"
Jul 18 01:37:17.617734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.617717    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b3343d-1504-49fe-958c-74001f3e8d8e" containerName="revoke-keys"
Jul 18 01:37:17.617734 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.617723    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b3343d-1504-49fe-958c-74001f3e8d8e" containerName="revoke-keys"
Jul 18 01:37:17.617867 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.617787    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b3343d-1504-49fe-958c-74001f3e8d8e" containerName="revoke-keys"
Jul 18 01:37:17.617867 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.617799    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="77620092-1cbc-486a-a296-5a7d54ec1bfa" containerName="maas-api"
Jul 18 01:37:17.622378 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.622359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"
Jul 18 01:37:17.627851 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.627820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"]
Jul 18 01:37:17.732037 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.731999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8\" (UID: \"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"
Jul 18 01:37:17.832584 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.832525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8\" (UID: \"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"
Jul 18 01:37:17.833218 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.833197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8\" (UID: \"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"
Jul 18 01:37:17.935232 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:17.935144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"
Jul 18 01:37:18.076434 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:18.076406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"]
Jul 18 01:37:18.077788 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:37:18.077757    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e129fbb_fb67_47ac_b2dd_fd70bdbfe20a.slice/crio-12c24f257f7f7fdb50b34dab7d09e99fed74ceb03cf3aa6fe60cb1750703a87c WatchSource:0}: Error finding container 12c24f257f7f7fdb50b34dab7d09e99fed74ceb03cf3aa6fe60cb1750703a87c: Status 404 returned error can't find the container with id 12c24f257f7f7fdb50b34dab7d09e99fed74ceb03cf3aa6fe60cb1750703a87c
Jul 18 01:37:18.737391 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:18.737351    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:37:18.967596 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:18.967555    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerID="e999efac9e61185e2569dda2a81e867e9fde389050bccf800bf2c5b38b8f28dc" exitCode=6
Jul 18 01:37:18.967809 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:18.967642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8" event={"ID":"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a","Type":"ContainerDied","Data":"e999efac9e61185e2569dda2a81e867e9fde389050bccf800bf2c5b38b8f28dc"}
Jul 18 01:37:18.967809 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:18.967695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8" event={"ID":"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a","Type":"ContainerStarted","Data":"12c24f257f7f7fdb50b34dab7d09e99fed74ceb03cf3aa6fe60cb1750703a87c"}
Jul 18 01:37:18.967938 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:18.967839    2563 scope.go:117] "RemoveContainer" containerID="e999efac9e61185e2569dda2a81e867e9fde389050bccf800bf2c5b38b8f28dc"
Jul 18 01:37:19.975724 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:19.975687    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerID="b8282b1b02f1422cd05ad9146bbaa8a501b0449494b2f2550ebfd8477affa32e" exitCode=6
Jul 18 01:37:19.976238 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:19.975763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8" event={"ID":"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a","Type":"ContainerDied","Data":"b8282b1b02f1422cd05ad9146bbaa8a501b0449494b2f2550ebfd8477affa32e"}
Jul 18 01:37:19.976238 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:19.975817    2563 scope.go:117] "RemoveContainer" containerID="e999efac9e61185e2569dda2a81e867e9fde389050bccf800bf2c5b38b8f28dc"
Jul 18 01:37:19.976238 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:19.976107    2563 scope.go:117] "RemoveContainer" containerID="b8282b1b02f1422cd05ad9146bbaa8a501b0449494b2f2550ebfd8477affa32e"
Jul 18 01:37:19.976403 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:19.976367    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8_odh-ai-gateway-infra(3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a"
Jul 18 01:37:20.984781 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:20.984753    2563 scope.go:117] "RemoveContainer" containerID="b8282b1b02f1422cd05ad9146bbaa8a501b0449494b2f2550ebfd8477affa32e"
Jul 18 01:37:20.985296 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:20.984946    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8_odh-ai-gateway-infra(3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a"
Jul 18 01:37:22.176250 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:22.176176    2563 scope.go:117] "RemoveContainer" containerID="49a6b5b97e65985ae9e58c1619f18ffb28800ebafeb8f3903e3992eec7da647c"
Jul 18 01:37:22.997960 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:22.997926    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerID="50a749b25db92f9468428f13d076a04a99bf6402b6fb4d6a2ce44d4e23d6225e" exitCode=6
Jul 18 01:37:22.998162 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:22.997995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65" event={"ID":"3d98aa27-4cfe-4d34-9202-deb5774c2533","Type":"ContainerDied","Data":"50a749b25db92f9468428f13d076a04a99bf6402b6fb4d6a2ce44d4e23d6225e"}
Jul 18 01:37:22.998162 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:22.998071    2563 scope.go:117] "RemoveContainer" containerID="49a6b5b97e65985ae9e58c1619f18ffb28800ebafeb8f3903e3992eec7da647c"
Jul 18 01:37:22.998374 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:22.998357    2563 scope.go:117] "RemoveContainer" containerID="50a749b25db92f9468428f13d076a04a99bf6402b6fb4d6a2ce44d4e23d6225e"
Jul 18 01:37:22.998638 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:22.998618    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4701d191-t6z65_odh-ai-gateway-infra(3d98aa27-4cfe-4d34-9202-deb5774c2533)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533"
Jul 18 01:37:24.024033 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:24.023988    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"]
Jul 18 01:37:24.151825 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:24.151799    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"
Jul 18 01:37:24.192945 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:24.192910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d98aa27-4cfe-4d34-9202-deb5774c2533-maas-api-service-ca\") pod \"3d98aa27-4cfe-4d34-9202-deb5774c2533\" (UID: \"3d98aa27-4cfe-4d34-9202-deb5774c2533\") "
Jul 18 01:37:24.193271 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:24.193238    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d98aa27-4cfe-4d34-9202-deb5774c2533-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d98aa27-4cfe-4d34-9202-deb5774c2533" (UID: "3d98aa27-4cfe-4d34-9202-deb5774c2533"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:37:24.294258 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:24.294171    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d98aa27-4cfe-4d34-9202-deb5774c2533-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:25.014362 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:25.014331    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"
Jul 18 01:37:25.014362 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:25.014348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65" event={"ID":"3d98aa27-4cfe-4d34-9202-deb5774c2533","Type":"ContainerDied","Data":"b3fa99c9d1f31b3fdf0918aa747f63740eee0397aa61b01eabfab6959adb72cf"}
Jul 18 01:37:25.014603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:25.014410    2563 scope.go:117] "RemoveContainer" containerID="50a749b25db92f9468428f13d076a04a99bf6402b6fb4d6a2ce44d4e23d6225e"
Jul 18 01:37:25.035580 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:25.035548    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"]
Jul 18 01:37:25.041337 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:25.041315    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4701d191-t6z65"]
Jul 18 01:37:26.176320 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:26.176289    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" path="/var/lib/kubelet/pods/3d98aa27-4cfe-4d34-9202-deb5774c2533/volumes"
Jul 18 01:37:31.084191 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084145    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"]
Jul 18 01:37:31.084727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:31.084727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084558    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:31.084727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084569    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:31.084727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084575    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:31.084727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084585    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:31.084727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084590    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:31.084727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084666    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:31.084727 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.084676    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:31.089081 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.089044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"
Jul 18 01:37:31.094641 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.094601    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"]
Jul 18 01:37:31.145583 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.145549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd7aca-61b2-45e7-ad38-7bedb680e5d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt\" (UID: \"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"
Jul 18 01:37:31.246127 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.246085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd7aca-61b2-45e7-ad38-7bedb680e5d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt\" (UID: \"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"
Jul 18 01:37:31.246794 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.246771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd7aca-61b2-45e7-ad38-7bedb680e5d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt\" (UID: \"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"
Jul 18 01:37:31.403433 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.403311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"
Jul 18 01:37:31.550387 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:31.550359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"]
Jul 18 01:37:31.551249 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:37:31.551220    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebd7aca_61b2_45e7_ad38_7bedb680e5d3.slice/crio-5b69ac676cc3a277118c68629540771b8fd6d2381bffc71acb6642cafbf18e4b WatchSource:0}: Error finding container 5b69ac676cc3a277118c68629540771b8fd6d2381bffc71acb6642cafbf18e4b: Status 404 returned error can't find the container with id 5b69ac676cc3a277118c68629540771b8fd6d2381bffc71acb6642cafbf18e4b
Jul 18 01:37:32.060892 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.060787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt" event={"ID":"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3","Type":"ContainerStarted","Data":"f5293764af5797893129c5da5c02edf623cc075318679dfeab3b841ffdfe6928"}
Jul 18 01:37:32.060892 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.060854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt" event={"ID":"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3","Type":"ContainerStarted","Data":"5b69ac676cc3a277118c68629540771b8fd6d2381bffc71acb6642cafbf18e4b"}
Jul 18 01:37:32.087128 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.087048    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt" podStartSLOduration=1.087029519 podStartE2EDuration="1.087029519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:37:32.085329587 +0000 UTC m=+3422.405306970" watchObservedRunningTime="2026-07-18 01:37:32.087029519 +0000 UTC m=+3422.407006901"
Jul 18 01:37:32.170313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.170276    2563 scope.go:117] "RemoveContainer" containerID="b8282b1b02f1422cd05ad9146bbaa8a501b0449494b2f2550ebfd8477affa32e"
Jul 18 01:37:32.699448 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.699416    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"]
Jul 18 01:37:32.699963 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.699941    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d98aa27-4cfe-4d34-9202-deb5774c2533" containerName="revoke-keys"
Jul 18 01:37:32.703071 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.703038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"
Jul 18 01:37:32.710074 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.710026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"]
Jul 18 01:37:32.758795 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.758756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01156a33-e9b6-4a3f-bc0e-9506158f0ec6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2\" (UID: \"01156a33-e9b6-4a3f-bc0e-9506158f0ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"
Jul 18 01:37:32.859875 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.859828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01156a33-e9b6-4a3f-bc0e-9506158f0ec6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2\" (UID: \"01156a33-e9b6-4a3f-bc0e-9506158f0ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"
Jul 18 01:37:32.860510 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:32.860488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01156a33-e9b6-4a3f-bc0e-9506158f0ec6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2\" (UID: \"01156a33-e9b6-4a3f-bc0e-9506158f0ec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"
Jul 18 01:37:33.014871 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:33.014763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"
Jul 18 01:37:33.070254 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:33.070213    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ebd7aca-61b2-45e7-ad38-7bedb680e5d3" containerID="f5293764af5797893129c5da5c02edf623cc075318679dfeab3b841ffdfe6928" exitCode=0
Jul 18 01:37:33.070485 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:33.070305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt" event={"ID":"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3","Type":"ContainerDied","Data":"f5293764af5797893129c5da5c02edf623cc075318679dfeab3b841ffdfe6928"}
Jul 18 01:37:33.072694 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:33.072666    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerID="e39aa1edbaa039947fcbcc472a66d99d23ad047d221ab5d144e2c56a850039f4" exitCode=6
Jul 18 01:37:33.072830 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:33.072726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8" event={"ID":"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a","Type":"ContainerDied","Data":"e39aa1edbaa039947fcbcc472a66d99d23ad047d221ab5d144e2c56a850039f4"}
Jul 18 01:37:33.072830 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:33.072800    2563 scope.go:117] "RemoveContainer" containerID="b8282b1b02f1422cd05ad9146bbaa8a501b0449494b2f2550ebfd8477affa32e"
Jul 18 01:37:33.073203 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:33.073181    2563 scope.go:117] "RemoveContainer" containerID="e39aa1edbaa039947fcbcc472a66d99d23ad047d221ab5d144e2c56a850039f4"
Jul 18 01:37:33.073564 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:33.073542    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8_odh-ai-gateway-infra(3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a"
Jul 18 01:37:33.177758 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:33.177437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"]
Jul 18 01:37:33.180685 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:37:33.180653    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01156a33_e9b6_4a3f_bc0e_9506158f0ec6.slice/crio-fec7fa9c7eeff2f9aa62a7e7176461665e6c973a6a45d0a8d2ea6d23bf9043dc WatchSource:0}: Error finding container fec7fa9c7eeff2f9aa62a7e7176461665e6c973a6a45d0a8d2ea6d23bf9043dc: Status 404 returned error can't find the container with id fec7fa9c7eeff2f9aa62a7e7176461665e6c973a6a45d0a8d2ea6d23bf9043dc
Jul 18 01:37:34.080736 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.080693    2563 generic.go:356] "Generic (PLEG): container finished" podID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerID="f684fa7754d7f714b137100639c49dcec92d1a35d9b76dc0d46fe193dfa4f52b" exitCode=6
Jul 18 01:37:34.080944 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.080782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2" event={"ID":"01156a33-e9b6-4a3f-bc0e-9506158f0ec6","Type":"ContainerDied","Data":"f684fa7754d7f714b137100639c49dcec92d1a35d9b76dc0d46fe193dfa4f52b"}
Jul 18 01:37:34.080944 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.080878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2" event={"ID":"01156a33-e9b6-4a3f-bc0e-9506158f0ec6","Type":"ContainerStarted","Data":"fec7fa9c7eeff2f9aa62a7e7176461665e6c973a6a45d0a8d2ea6d23bf9043dc"}
Jul 18 01:37:34.081084 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.080989    2563 scope.go:117] "RemoveContainer" containerID="f684fa7754d7f714b137100639c49dcec92d1a35d9b76dc0d46fe193dfa4f52b"
Jul 18 01:37:34.113033 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.112999    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"]
Jul 18 01:37:34.240973 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.240943    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"
Jul 18 01:37:34.264533 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.264502    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"
Jul 18 01:37:34.272209 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.272180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd7aca-61b2-45e7-ad38-7bedb680e5d3-maas-api-service-ca\") pod \"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3\" (UID: \"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3\") "
Jul 18 01:37:34.272357 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.272231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a-maas-api-service-ca\") pod \"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a\" (UID: \"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a\") "
Jul 18 01:37:34.272548 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.272524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebd7aca-61b2-45e7-ad38-7bedb680e5d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ebd7aca-61b2-45e7-ad38-7bedb680e5d3" (UID: "0ebd7aca-61b2-45e7-ad38-7bedb680e5d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:37:34.272650 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.272615    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" (UID: "3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:37:34.373596 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.373560    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebd7aca-61b2-45e7-ad38-7bedb680e5d3-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:34.373596 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:34.373591    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:35.090576 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.090497    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt"
Jul 18 01:37:35.090576 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.090537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt" event={"ID":"0ebd7aca-61b2-45e7-ad38-7bedb680e5d3","Type":"ContainerDied","Data":"5b69ac676cc3a277118c68629540771b8fd6d2381bffc71acb6642cafbf18e4b"}
Jul 18 01:37:35.090776 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.090589    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b69ac676cc3a277118c68629540771b8fd6d2381bffc71acb6642cafbf18e4b"
Jul 18 01:37:35.092485 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.092459    2563 generic.go:356] "Generic (PLEG): container finished" podID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerID="cd9d45480f7099897fc13cb666a1f139728dd58d022540d2a420b33086f46968" exitCode=6
Jul 18 01:37:35.092647 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.092546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2" event={"ID":"01156a33-e9b6-4a3f-bc0e-9506158f0ec6","Type":"ContainerDied","Data":"cd9d45480f7099897fc13cb666a1f139728dd58d022540d2a420b33086f46968"}
Jul 18 01:37:35.092647 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.092613    2563 scope.go:117] "RemoveContainer" containerID="f684fa7754d7f714b137100639c49dcec92d1a35d9b76dc0d46fe193dfa4f52b"
Jul 18 01:37:35.092798 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.092745    2563 scope.go:117] "RemoveContainer" containerID="cd9d45480f7099897fc13cb666a1f139728dd58d022540d2a420b33086f46968"
Jul 18 01:37:35.092986 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:35.092966    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2_odh-ai-gateway-infra(01156a33-e9b6-4a3f-bc0e-9506158f0ec6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6"
Jul 18 01:37:35.093971 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.093957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"
Jul 18 01:37:35.094083 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.094003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8" event={"ID":"3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a","Type":"ContainerDied","Data":"12c24f257f7f7fdb50b34dab7d09e99fed74ceb03cf3aa6fe60cb1750703a87c"}
Jul 18 01:37:35.105505 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.105485    2563 scope.go:117] "RemoveContainer" containerID="e39aa1edbaa039947fcbcc472a66d99d23ad047d221ab5d144e2c56a850039f4"
Jul 18 01:37:35.129420 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.129389    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"]
Jul 18 01:37:35.133002 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:35.132975    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf4f6c8d-jd7n8"]
Jul 18 01:37:36.101770 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:36.101732    2563 scope.go:117] "RemoveContainer" containerID="cd9d45480f7099897fc13cb666a1f139728dd58d022540d2a420b33086f46968"
Jul 18 01:37:36.102271 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:36.101974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2_odh-ai-gateway-infra(01156a33-e9b6-4a3f-bc0e-9506158f0ec6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6"
Jul 18 01:37:36.176709 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:36.176673    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" path="/var/lib/kubelet/pods/3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a/volumes"
Jul 18 01:37:41.488344 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.487951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"]
Jul 18 01:37:41.488842 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.488642    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h" podUID="2a8ff4b1-b150-4ca9-915b-6166c729bad1" containerName="maas-api" containerID="cri-o://8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173" gracePeriod=30
Jul 18 01:37:41.739744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.739682    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:37:41.833100 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.833064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-494fz\" (UniqueName: \"kubernetes.io/projected/2a8ff4b1-b150-4ca9-915b-6166c729bad1-kube-api-access-494fz\") pod \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") "
Jul 18 01:37:41.833272 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.833108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls\") pod \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") "
Jul 18 01:37:41.833272 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.833153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ff4b1-b150-4ca9-915b-6166c729bad1-openshift-service-ca\") pod \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\" (UID: \"2a8ff4b1-b150-4ca9-915b-6166c729bad1\") "
Jul 18 01:37:41.833559 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.833529    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8ff4b1-b150-4ca9-915b-6166c729bad1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a8ff4b1-b150-4ca9-915b-6166c729bad1" (UID: "2a8ff4b1-b150-4ca9-915b-6166c729bad1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:37:41.835199 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.835171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8ff4b1-b150-4ca9-915b-6166c729bad1-kube-api-access-494fz" (OuterVolumeSpecName: "kube-api-access-494fz") pod "2a8ff4b1-b150-4ca9-915b-6166c729bad1" (UID: "2a8ff4b1-b150-4ca9-915b-6166c729bad1"). InnerVolumeSpecName "kube-api-access-494fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 01:37:41.835282 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.835236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a8ff4b1-b150-4ca9-915b-6166c729bad1" (UID: "2a8ff4b1-b150-4ca9-915b-6166c729bad1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 01:37:41.934259 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.934222    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-494fz\" (UniqueName: \"kubernetes.io/projected/2a8ff4b1-b150-4ca9-915b-6166c729bad1-kube-api-access-494fz\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:41.934259 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.934261    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8ff4b1-b150-4ca9-915b-6166c729bad1-maas-api-tls\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:41.934465 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:41.934277    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8ff4b1-b150-4ca9-915b-6166c729bad1-openshift-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:42.145095 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.144986    2563 generic.go:356] "Generic (PLEG): container finished" podID="2a8ff4b1-b150-4ca9-915b-6166c729bad1" containerID="8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173" exitCode=0
Jul 18 01:37:42.145095 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.145029    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"
Jul 18 01:37:42.145346 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.145090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h" event={"ID":"2a8ff4b1-b150-4ca9-915b-6166c729bad1","Type":"ContainerDied","Data":"8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173"}
Jul 18 01:37:42.145346 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.145151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h" event={"ID":"2a8ff4b1-b150-4ca9-915b-6166c729bad1","Type":"ContainerDied","Data":"ecfb641e083eec8c7f7c7e6d8ac7d349bf571386924f2511996eadd4e4a9ccba"}
Jul 18 01:37:42.145346 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.145183    2563 scope.go:117] "RemoveContainer" containerID="8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173"
Jul 18 01:37:42.155530 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.155514    2563 scope.go:117] "RemoveContainer" containerID="8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173"
Jul 18 01:37:42.155781 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:42.155761    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173\": container with ID starting with 8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173 not found: ID does not exist" containerID="8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173"
Jul 18 01:37:42.155842 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.155789    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173"} err="failed to get container status \"8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173\": rpc error: code = NotFound desc = could not find container \"8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173\": container with ID starting with 8edcdc66184d6942567115ce7579132fb6e5344e628270e213f55f436dfdc173 not found: ID does not exist"
Jul 18 01:37:42.167408 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.167387    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"]
Jul 18 01:37:42.176821 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:42.176801    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0f7fdc-664d4d64cf-l528h"]
Jul 18 01:37:44.175730 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:44.175690    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8ff4b1-b150-4ca9-915b-6166c729bad1" path="/var/lib/kubelet/pods/2a8ff4b1-b150-4ca9-915b-6166c729bad1/volumes"
Jul 18 01:37:44.640452 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:44.640416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2gs9p"]
Jul 18 01:37:47.169943 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:47.169904    2563 scope.go:117] "RemoveContainer" containerID="cd9d45480f7099897fc13cb666a1f139728dd58d022540d2a420b33086f46968"
Jul 18 01:37:48.189224 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:48.189141    2563 generic.go:356] "Generic (PLEG): container finished" podID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerID="28492f22fd60ae6789abd5d4a861f08095c640a2f44f827eb5b9532c76bbb916" exitCode=6
Jul 18 01:37:48.189224 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:48.189191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2" event={"ID":"01156a33-e9b6-4a3f-bc0e-9506158f0ec6","Type":"ContainerDied","Data":"28492f22fd60ae6789abd5d4a861f08095c640a2f44f827eb5b9532c76bbb916"}
Jul 18 01:37:48.189661 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:48.189238    2563 scope.go:117] "RemoveContainer" containerID="cd9d45480f7099897fc13cb666a1f139728dd58d022540d2a420b33086f46968"
Jul 18 01:37:48.189661 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:48.189487    2563 scope.go:117] "RemoveContainer" containerID="28492f22fd60ae6789abd5d4a861f08095c640a2f44f827eb5b9532c76bbb916"
Jul 18 01:37:48.189745 ip-10-0-143-142 kubenswrapper[2563]: E0718 01:37:48.189693    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2_odh-ai-gateway-infra(01156a33-e9b6-4a3f-bc0e-9506158f0ec6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6"
Jul 18 01:37:49.214283 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:49.214248    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"]
Jul 18 01:37:49.351289 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:49.351260    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"
Jul 18 01:37:49.398888 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:49.398860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01156a33-e9b6-4a3f-bc0e-9506158f0ec6-maas-api-service-ca\") pod \"01156a33-e9b6-4a3f-bc0e-9506158f0ec6\" (UID: \"01156a33-e9b6-4a3f-bc0e-9506158f0ec6\") "
Jul 18 01:37:49.399249 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:49.399226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01156a33-e9b6-4a3f-bc0e-9506158f0ec6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01156a33-e9b6-4a3f-bc0e-9506158f0ec6" (UID: "01156a33-e9b6-4a3f-bc0e-9506158f0ec6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 01:37:49.500404 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:49.500322    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01156a33-e9b6-4a3f-bc0e-9506158f0ec6-maas-api-service-ca\") on node \"ip-10-0-143-142.ec2.internal\" DevicePath \"\""
Jul 18 01:37:50.203679 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:50.203644    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"
Jul 18 01:37:50.203864 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:50.203685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2" event={"ID":"01156a33-e9b6-4a3f-bc0e-9506158f0ec6","Type":"ContainerDied","Data":"fec7fa9c7eeff2f9aa62a7e7176461665e6c973a6a45d0a8d2ea6d23bf9043dc"}
Jul 18 01:37:50.203864 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:50.203745    2563 scope.go:117] "RemoveContainer" containerID="28492f22fd60ae6789abd5d4a861f08095c640a2f44f827eb5b9532c76bbb916"
Jul 18 01:37:50.220953 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:50.220926    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"]
Jul 18 01:37:50.224887 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:50.224859    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4e26d7cc-nfzt2"]
Jul 18 01:37:52.175708 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:37:52.175623    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" path="/var/lib/kubelet/pods/01156a33-e9b6-4a3f-bc0e-9506158f0ec6/volumes"
Jul 18 01:38:02.544068 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:02.544034    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jgrhr_c8b41baa-e1f0-4ccc-9fa9-00e46c4af052/manager/0.log"
Jul 18 01:38:02.663961 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:02.663925    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fmcn7_444186e3-c146-4024-b7fb-e99ecbe93a58/manager/0.log"
Jul 18 01:38:02.782636 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:02.782595    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bc99686f4-z7dtd_aca6fde9-4e41-4542-b5c0-39461d3782f0/manager/0.log"
Jul 18 01:38:02.888106 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:02.888016    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2r5ds_50b94f72-c9ff-417d-a757-17c2f33d642d/server/0.log"
Jul 18 01:38:03.236271 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:03.236237    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85c6dd979f-nzm74_28a1cf60-87ae-450c-b575-0c7bb88cf938/copy-manifests/0.log"
Jul 18 01:38:03.242785 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:03.242756    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85c6dd979f-nzm74_28a1cf60-87ae-450c-b575-0c7bb88cf938/manager/0.log"
Jul 18 01:38:03.766832 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:03.766784    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f9fb8f54-4hqms_eaa1b620-c2fa-42d5-a241-4106d914b329/maas-api/0.log"
Jul 18 01:38:03.873025 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:03.872996    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738940-h58q4_da4650b9-dd68-43a7-9308-febaaea81688/cleanup/0.log"
Jul 18 01:38:04.083620 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:04.083530    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738970-2pkm2_29c42ac8-efeb-4713-84cc-0b6463c9f80f/cleanup/0.log"
Jul 18 01:38:04.393367 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:04.393286    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-0192d7-csqtd_4019fb29-565b-4347-9e83-ae2e1f398447/revoke-keys/0.log"
Jul 18 01:38:04.498673 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:04.498642    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-232dbd-zsgb9_7cbd8ed3-b649-4fda-93c3-00a4eb88dca9/revoke-keys/0.log"
Jul 18 01:38:04.603976 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:04.603937    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-390947-qwlhg_1e251588-1e23-41a8-8af4-eb1cf199b26e/revoke-keys/0.log"
Jul 18 01:38:04.715190 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:04.715162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-0a3406-lrvsf_4775064f-5f91-47af-a762-288e0408e1ca/revoke-keys/0.log"
Jul 18 01:38:05.039313 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:05.039212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-ad1b67-6rh6h_94b3343d-1504-49fe-958c-74001f3e8d8e/revoke-keys/0.log"
Jul 18 01:38:05.145619 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:05.145583    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-0f7fdc-74nmt_0ebd7aca-61b2-45e7-ad38-7bedb680e5d3/revoke-keys/0.log"
Jul 18 01:38:05.986617 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:05.986584    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n_7b0e438c-9503-437a-9114-aad368d08d6b/util/0.log"
Jul 18 01:38:05.992440 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:05.992405    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n_7b0e438c-9503-437a-9114-aad368d08d6b/pull/0.log"
Jul 18 01:38:05.997812 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:05.997789    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n_7b0e438c-9503-437a-9114-aad368d08d6b/extract/0.log"
Jul 18 01:38:06.103846 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.103815    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2_9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae/util/0.log"
Jul 18 01:38:06.109987 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.109958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2_9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae/pull/0.log"
Jul 18 01:38:06.115325 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.115303    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2_9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae/extract/0.log"
Jul 18 01:38:06.220554 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.220522    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8_f45e873e-7fbe-48bd-bd1a-0f25d09e6af5/util/0.log"
Jul 18 01:38:06.226383 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.226350    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8_f45e873e-7fbe-48bd-bd1a-0f25d09e6af5/pull/0.log"
Jul 18 01:38:06.232334 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.232310    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8_f45e873e-7fbe-48bd-bd1a-0f25d09e6af5/extract/0.log"
Jul 18 01:38:06.337677 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.337587    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt_383d5a6b-b7d6-4230-bb31-b53a88a7978e/extract/0.log"
Jul 18 01:38:06.343264 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.343240    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt_383d5a6b-b7d6-4230-bb31-b53a88a7978e/util/0.log"
Jul 18 01:38:06.349516 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.349483    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt_383d5a6b-b7d6-4230-bb31-b53a88a7978e/pull/0.log"
Jul 18 01:38:06.578247 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.578211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-w28jc_a1faae7b-f155-4cba-93da-dd68e15bb2ba/manager/0.log"
Jul 18 01:38:06.684029 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.683938    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-r5pfv_8f62bc9e-a655-41b2-8a0a-19928ddf5013/manager/0.log"
Jul 18 01:38:06.793769 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.793731    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4nv6z_669e83df-f790-4e7c-acdd-da69464bc03f/kuadrant-console-plugin/0.log"
Jul 18 01:38:06.927876 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:06.927843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7mgl5_d45e6e69-9468-4381-9e68-1a3c8f895e47/registry-server/0.log"
Jul 18 01:38:07.050873 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:07.050832    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-zfbfh_80dd2b8c-bf37-4c84-8579-4bddc6d9070f/manager/0.log"
Jul 18 01:38:07.154648 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:07.154616    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2gs9p_f85b8312-3894-4a9e-9a94-10c880a177ea/limitador/0.log"
Jul 18 01:38:07.272598 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:07.272564    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x2nfq_5d634b99-8329-4306-b7a4-4987bc6d61fd/manager/0.log"
Jul 18 01:38:07.926477 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:07.926445    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f655d47ff-phrzs_fbe113b3-f563-4819-85db-43159544559b/kube-auth-proxy/0.log"
Jul 18 01:38:08.282767 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:08.282730    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-z5wd9_1d8b1761-aa3c-4913-a140-af8dbe0ccb82/payload-processing/0.log"
Jul 18 01:38:09.038521 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:09.038489    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwzznq_122d0d0b-1459-4f74-aac5-a51282cd3571/main/0.log"
Jul 18 01:38:14.490185 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490151    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gb6px/must-gather-hlkg9"]
Jul 18 01:38:14.490670 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490651    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490673    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490682    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490687    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490694    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490699    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490708    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8ff4b1-b150-4ca9-915b-6166c729bad1" containerName="maas-api"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490713    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8ff4b1-b150-4ca9-915b-6166c729bad1" containerName="maas-api"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490719    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ebd7aca-61b2-45e7-ad38-7bedb680e5d3" containerName="revoke-keys"
Jul 18 01:38:14.490723 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490724    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ebd7aca-61b2-45e7-ad38-7bedb680e5d3" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490737    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490742    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490750    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490755    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490813    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490821    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8ff4b1-b150-4ca9-915b-6166c729bad1" containerName="maas-api"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490828    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490835    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490841    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ebd7aca-61b2-45e7-ad38-7bedb680e5d3" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490846    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490851    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e129fbb-fb67-47ac-b2dd-fd70bdbfe20a" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490910    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490916    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.490993 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.490970    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01156a33-e9b6-4a3f-bc0e-9506158f0ec6" containerName="revoke-keys"
Jul 18 01:38:14.494097 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.494079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gb6px/must-gather-hlkg9"
Jul 18 01:38:14.496649 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.496620    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gb6px\"/\"kube-root-ca.crt\""
Jul 18 01:38:14.496769 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.496661    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gb6px\"/\"openshift-service-ca.crt\""
Jul 18 01:38:14.502584 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.502554    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gb6px/must-gather-hlkg9"]
Jul 18 01:38:14.618547 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.618509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s47w\" (UniqueName: \"kubernetes.io/projected/23f68e63-bc37-421c-b497-6bc123710bea-kube-api-access-6s47w\") pod \"must-gather-hlkg9\" (UID: \"23f68e63-bc37-421c-b497-6bc123710bea\") " pod="openshift-must-gather-gb6px/must-gather-hlkg9"
Jul 18 01:38:14.618731 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.618580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23f68e63-bc37-421c-b497-6bc123710bea-must-gather-output\") pod \"must-gather-hlkg9\" (UID: \"23f68e63-bc37-421c-b497-6bc123710bea\") " pod="openshift-must-gather-gb6px/must-gather-hlkg9"
Jul 18 01:38:14.719926 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.719886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s47w\" (UniqueName: \"kubernetes.io/projected/23f68e63-bc37-421c-b497-6bc123710bea-kube-api-access-6s47w\") pod \"must-gather-hlkg9\" (UID: \"23f68e63-bc37-421c-b497-6bc123710bea\") " pod="openshift-must-gather-gb6px/must-gather-hlkg9"
Jul 18 01:38:14.720136 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.719980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23f68e63-bc37-421c-b497-6bc123710bea-must-gather-output\") pod \"must-gather-hlkg9\" (UID: \"23f68e63-bc37-421c-b497-6bc123710bea\") " pod="openshift-must-gather-gb6px/must-gather-hlkg9"
Jul 18 01:38:14.720392 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.720369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23f68e63-bc37-421c-b497-6bc123710bea-must-gather-output\") pod \"must-gather-hlkg9\" (UID: \"23f68e63-bc37-421c-b497-6bc123710bea\") " pod="openshift-must-gather-gb6px/must-gather-hlkg9"
Jul 18 01:38:14.727748 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.727711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s47w\" (UniqueName: \"kubernetes.io/projected/23f68e63-bc37-421c-b497-6bc123710bea-kube-api-access-6s47w\") pod \"must-gather-hlkg9\" (UID: \"23f68e63-bc37-421c-b497-6bc123710bea\") " pod="openshift-must-gather-gb6px/must-gather-hlkg9"
Jul 18 01:38:14.804561 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.804474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gb6px/must-gather-hlkg9"
Jul 18 01:38:14.939642 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:14.939615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gb6px/must-gather-hlkg9"]
Jul 18 01:38:14.941296 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:38:14.941269    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23f68e63_bc37_421c_b497_6bc123710bea.slice/crio-90229eb1cca554ad64efbb14b276fc760379f6ece35da0e958bffb06ddd0a893 WatchSource:0}: Error finding container 90229eb1cca554ad64efbb14b276fc760379f6ece35da0e958bffb06ddd0a893: Status 404 returned error can't find the container with id 90229eb1cca554ad64efbb14b276fc760379f6ece35da0e958bffb06ddd0a893
Jul 18 01:38:15.356165 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:15.356121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gb6px/must-gather-hlkg9" event={"ID":"23f68e63-bc37-421c-b497-6bc123710bea","Type":"ContainerStarted","Data":"90229eb1cca554ad64efbb14b276fc760379f6ece35da0e958bffb06ddd0a893"}
Jul 18 01:38:16.367913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:16.367408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gb6px/must-gather-hlkg9" event={"ID":"23f68e63-bc37-421c-b497-6bc123710bea","Type":"ContainerStarted","Data":"d6ad8481d62e794aacdd6e1246503052361f322e89796ac8e3a6110b7231db8b"}
Jul 18 01:38:16.367913 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:16.367463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gb6px/must-gather-hlkg9" event={"ID":"23f68e63-bc37-421c-b497-6bc123710bea","Type":"ContainerStarted","Data":"a5ea08c6c078c244ca3444368d653ef44a22e27b29e342cdf2b7ee5540735a82"}
Jul 18 01:38:16.387198 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:16.387126    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gb6px/must-gather-hlkg9" podStartSLOduration=1.454138483 podStartE2EDuration="2.387103895s" totalImagesPullingTime="932.965412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:38:16.381449342 +0000 UTC m=+3466.701426725" watchObservedRunningTime="2026-07-18 01:38:16.387103895 +0000 UTC m=+3466.707081279"
Jul 18 01:38:17.446371 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:17.446338    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g5fq8_2a5b6fba-2c56-4526-9e0f-21ca35cba6f2/global-pull-secret-syncer/0.log"
Jul 18 01:38:17.621638 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:17.621597    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ghhg9_139e9ed7-f820-45ad-ac30-a24de9377afe/konnectivity-agent/0.log"
Jul 18 01:38:17.719576 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:17.719494    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-142.ec2.internal_f41603fa42d7d0e66deed9946e4a9d78/haproxy/0.log"
Jul 18 01:38:22.795078 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:22.792902    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n_7b0e438c-9503-437a-9114-aad368d08d6b/extract/0.log"
Jul 18 01:38:22.829080 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:22.827923    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n_7b0e438c-9503-437a-9114-aad368d08d6b/util/0.log"
Jul 18 01:38:22.856685 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:22.856232    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dxp2n_7b0e438c-9503-437a-9114-aad368d08d6b/pull/0.log"
Jul 18 01:38:22.897073 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:22.897024    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2_9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae/extract/0.log"
Jul 18 01:38:22.916344 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:22.916310    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2_9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae/util/0.log"
Jul 18 01:38:22.937878 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:22.937843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w57x2_9306e0e7-d6e3-42f6-98b2-e1f6316fc7ae/pull/0.log"
Jul 18 01:38:22.962390 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:22.962348    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8_f45e873e-7fbe-48bd-bd1a-0f25d09e6af5/extract/0.log"
Jul 18 01:38:22.982585 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:22.982553    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8_f45e873e-7fbe-48bd-bd1a-0f25d09e6af5/util/0.log"
Jul 18 01:38:23.004420 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.004374    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73drjn8_f45e873e-7fbe-48bd-bd1a-0f25d09e6af5/pull/0.log"
Jul 18 01:38:23.029458 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.029425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt_383d5a6b-b7d6-4230-bb31-b53a88a7978e/extract/0.log"
Jul 18 01:38:23.049394 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.049321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt_383d5a6b-b7d6-4230-bb31-b53a88a7978e/util/0.log"
Jul 18 01:38:23.073844 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.073812    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qtflt_383d5a6b-b7d6-4230-bb31-b53a88a7978e/pull/0.log"
Jul 18 01:38:23.702384 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.702346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-w28jc_a1faae7b-f155-4cba-93da-dd68e15bb2ba/manager/0.log"
Jul 18 01:38:23.730426 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.730393    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-r5pfv_8f62bc9e-a655-41b2-8a0a-19928ddf5013/manager/0.log"
Jul 18 01:38:23.756927 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.756892    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4nv6z_669e83df-f790-4e7c-acdd-da69464bc03f/kuadrant-console-plugin/0.log"
Jul 18 01:38:23.805321 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.805292    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7mgl5_d45e6e69-9468-4381-9e68-1a3c8f895e47/registry-server/0.log"
Jul 18 01:38:23.948776 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.948736    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-zfbfh_80dd2b8c-bf37-4c84-8579-4bddc6d9070f/manager/0.log"
Jul 18 01:38:23.969486 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:23.969380    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2gs9p_f85b8312-3894-4a9e-9a94-10c880a177ea/limitador/0.log"
Jul 18 01:38:24.066081 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:24.066017    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x2nfq_5d634b99-8329-4306-b7a4-4987bc6d61fd/manager/0.log"
Jul 18 01:38:25.979211 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:25.979178    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gbdl_1ccc27d1-3918-479e-9da2-ae70606fab11/node-exporter/0.log"
Jul 18 01:38:25.997590 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:25.997561    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gbdl_1ccc27d1-3918-479e-9da2-ae70606fab11/kube-rbac-proxy/0.log"
Jul 18 01:38:26.017543 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:26.017509    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gbdl_1ccc27d1-3918-479e-9da2-ae70606fab11/init-textfile/0.log"
Jul 18 01:38:26.418709 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:26.418620    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5ddd96bc-l977r_0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54/telemeter-client/0.log"
Jul 18 01:38:26.439567 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:26.439536    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5ddd96bc-l977r_0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54/reload/0.log"
Jul 18 01:38:26.458618 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:26.458591    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d5ddd96bc-l977r_0b76f8fa-dcb3-4dfa-bfee-bec5ac8c7a54/kube-rbac-proxy/0.log"
Jul 18 01:38:27.401356 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.401316    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"]
Jul 18 01:38:27.409543 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.409512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.413798 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.413769    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gb6px\"/\"default-dockercfg-hr289\""
Jul 18 01:38:27.414944 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.414917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"]
Jul 18 01:38:27.554537 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.554494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-proc\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.554733 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.554641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-lib-modules\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.554733 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.554703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhk5\" (UniqueName: \"kubernetes.io/projected/44287cc2-ab2c-4924-a8dc-17c374de7e5a-kube-api-access-fjhk5\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.554872 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.554751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-podres\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.554872 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.554790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-sys\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-podres\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-sys\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656328 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-proc\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-lib-modules\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-podres\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhk5\" (UniqueName: \"kubernetes.io/projected/44287cc2-ab2c-4924-a8dc-17c374de7e5a-kube-api-access-fjhk5\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-lib-modules\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-proc\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.656603 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.656454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44287cc2-ab2c-4924-a8dc-17c374de7e5a-sys\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.665000 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.664969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhk5\" (UniqueName: \"kubernetes.io/projected/44287cc2-ab2c-4924-a8dc-17c374de7e5a-kube-api-access-fjhk5\") pod \"perf-node-gather-daemonset-4t2d5\" (UID: \"44287cc2-ab2c-4924-a8dc-17c374de7e5a\") " pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.725712 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.725668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:27.924482 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:27.924414    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"]
Jul 18 01:38:27.928158 ip-10-0-143-142 kubenswrapper[2563]: W0718 01:38:27.928116    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod44287cc2_ab2c_4924_a8dc_17c374de7e5a.slice/crio-cb40d756a195ec910462587eba2aadca7b9eec03c7fdc4b669de490bab1e2774 WatchSource:0}: Error finding container cb40d756a195ec910462587eba2aadca7b9eec03c7fdc4b669de490bab1e2774: Status 404 returned error can't find the container with id cb40d756a195ec910462587eba2aadca7b9eec03c7fdc4b669de490bab1e2774
Jul 18 01:38:28.502712 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:28.502666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5" event={"ID":"44287cc2-ab2c-4924-a8dc-17c374de7e5a","Type":"ContainerStarted","Data":"f63cb768ea0cee8e6a60338f5a081132849eda0ebf45975fef3217288e230ad5"}
Jul 18 01:38:28.503205 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:28.502761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:28.503205 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:28.502818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5" event={"ID":"44287cc2-ab2c-4924-a8dc-17c374de7e5a","Type":"ContainerStarted","Data":"cb40d756a195ec910462587eba2aadca7b9eec03c7fdc4b669de490bab1e2774"}
Jul 18 01:38:28.520134 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:28.520046    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5" podStartSLOduration=1.520022585 podStartE2EDuration="1.520022585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:38:28.516673319 +0000 UTC m=+3478.836650723" watchObservedRunningTime="2026-07-18 01:38:28.520022585 +0000 UTC m=+3478.839999969"
Jul 18 01:38:28.807911 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:28.807826    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ccbc64d7b-mnttf_f3d2adf1-0225-42be-8ba7-c38f60f95234/console/0.log"
Jul 18 01:38:28.854000 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:28.853957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-9gd7s_9f145695-b0d2-48bd-bf15-5beb1a409523/download-server/0.log"
Jul 18 01:38:30.454215 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:30.454184    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jzfr6_ab019bac-1682-4e2c-9167-894dc2ad5cb1/dns/0.log"
Jul 18 01:38:30.475763 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:30.475723    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jzfr6_ab019bac-1682-4e2c-9167-894dc2ad5cb1/kube-rbac-proxy/0.log"
Jul 18 01:38:30.598560 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:30.598519    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgnlq_6c200b84-577b-45d8-a6b6-e715ce2d4262/dns-node-resolver/0.log"
Jul 18 01:38:31.232795 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:31.232760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tkd94_beeaa768-6025-4191-9306-1c75abdd1477/node-ca/0.log"
Jul 18 01:38:32.334394 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:32.334361    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f655d47ff-phrzs_fbe113b3-f563-4819-85db-43159544559b/kube-auth-proxy/0.log"
Jul 18 01:38:32.457693 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:32.457662    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-z5wd9_1d8b1761-aa3c-4913-a140-af8dbe0ccb82/payload-processing/0.log"
Jul 18 01:38:33.194503 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:33.194470    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7sw4_112acebf-c853-4157-9d00-93c27387440f/serve-healthcheck-canary/0.log"
Jul 18 01:38:33.690541 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:33.690509    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4zgt4_82f6fc6a-2500-4c10-860c-2b2699f81669/kube-rbac-proxy/0.log"
Jul 18 01:38:33.709433 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:33.709402    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4zgt4_82f6fc6a-2500-4c10-860c-2b2699f81669/exporter/0.log"
Jul 18 01:38:33.729461 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:33.729428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4zgt4_82f6fc6a-2500-4c10-860c-2b2699f81669/extractor/0.log"
Jul 18 01:38:35.529785 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:35.529759    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gb6px/perf-node-gather-daemonset-4t2d5"
Jul 18 01:38:36.070663 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:36.070631    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jgrhr_c8b41baa-e1f0-4ccc-9fa9-00e46c4af052/manager/0.log"
Jul 18 01:38:36.347542 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:36.347454    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fmcn7_444186e3-c146-4024-b7fb-e99ecbe93a58/manager/0.log"
Jul 18 01:38:36.569999 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:36.569957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bc99686f4-z7dtd_aca6fde9-4e41-4542-b5c0-39461d3782f0/manager/0.log"
Jul 18 01:38:36.594744 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:36.594712    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2r5ds_50b94f72-c9ff-417d-a757-17c2f33d642d/server/0.log"
Jul 18 01:38:36.783852 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:36.783818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85c6dd979f-nzm74_28a1cf60-87ae-450c-b575-0c7bb88cf938/manager/0.log"
Jul 18 01:38:36.803867 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:36.803840    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85c6dd979f-nzm74_28a1cf60-87ae-450c-b575-0c7bb88cf938/copy-manifests/0.log"
Jul 18 01:38:38.344645 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:38.344596    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-m2mdr_3e7136b8-a288-4ba9-a2c3-17457bcf6e5b/openshift-lws-operator/0.log"
Jul 18 01:38:38.444602 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:38.444572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f655d47ff-phrzs_fbe113b3-f563-4819-85db-43159544559b/kube-auth-proxy/0.log"
Jul 18 01:38:38.547832 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:38.547800    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-z5wd9_1d8b1761-aa3c-4913-a140-af8dbe0ccb82/payload-processing/0.log"
Jul 18 01:38:39.905504 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:39.905417    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f655d47ff-phrzs_fbe113b3-f563-4819-85db-43159544559b/kube-auth-proxy/0.log"
Jul 18 01:38:40.016277 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:40.016234    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-z5wd9_1d8b1761-aa3c-4913-a140-af8dbe0ccb82/payload-processing/0.log"
Jul 18 01:38:43.348800 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:43.348762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-hlx4n_6123a428-e436-4a58-a29d-e3f9c671bbc1/migrator/0.log"
Jul 18 01:38:43.367354 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:43.367319    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-hlx4n_6123a428-e436-4a58-a29d-e3f9c671bbc1/graceful-termination/0.log"
Jul 18 01:38:44.749931 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:44.749892    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6zmpk_c79b6feb-e6d8-41e4-a8b4-15ae9ae1d70e/kube-multus/0.log"
Jul 18 01:38:44.942511 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:44.942477    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k4lcb_02374191-e591-4213-b783-3fc5d47e9625/kube-multus-additional-cni-plugins/0.log"
Jul 18 01:38:44.963135 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:44.963094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k4lcb_02374191-e591-4213-b783-3fc5d47e9625/egress-router-binary-copy/0.log"
Jul 18 01:38:44.981846 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:44.981813    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k4lcb_02374191-e591-4213-b783-3fc5d47e9625/cni-plugins/0.log"
Jul 18 01:38:45.002952 ip-10-0-143-142 kubenswrapper[2563]: I0718 01:38:45.002879    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k4lcb_02374191-e591-4213-b783-3fc5d47e9625/bond-cni-plugin/0.log"