Sep 09 06:51:15.559011 ip-10-0-142-122 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 06:51:15.905934 ip-10-0-142-122 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 06:51:15.905934 ip-10-0-142-122 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 06:51:15.905934 ip-10-0-142-122 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 06:51:15.905934 ip-10-0-142-122 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 06:51:15.905934 ip-10-0-142-122 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 06:51:15.907432 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.907349    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 06:51:15.910094 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910079    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 06:51:15.910094 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910094    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910098    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910102    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910106    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910109    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910113    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910117    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910121    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910123    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910126    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910128    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910131    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910134    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910136    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910139    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910141    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910144    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910146    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910148    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 06:51:15.910155 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910151    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910154    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910161    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910164    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910168    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910171    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910173    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910176    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910178    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910181    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910184    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910186    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910189    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910191    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910194    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910196    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910199    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910201    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910204    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 06:51:15.910604 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910206    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910209    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910211    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910214    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910216    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910219    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910221    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910224    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910227    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910229    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910232    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910235    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910237    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910240    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910243    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910246    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910249    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910252    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910254    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910256    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 06:51:15.911092 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910260    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910264    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910267    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910270    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910273    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910275    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910278    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910282    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910284    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910287    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910289    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910292    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910295    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910298    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910301    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910304    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910307    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910310    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910315    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910317    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 06:51:15.911616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910320    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910322    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910325    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910328    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910330    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910333    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910335    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910338    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910341    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910344    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910346    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910350    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910353    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910357    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910359    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910362    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910365    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910367    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910370    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 06:51:15.912110 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.910372    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911058    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911065    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911068    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911071    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911074    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911077    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911079    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911084    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911089    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911093    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911098    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911101    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911104    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911107    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911110    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911113    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911115    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911118    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 06:51:15.912570 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911120    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911123    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911126    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911129    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911131    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911134    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911136    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911139    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911141    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911144    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911147    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911149    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911152    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911155    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911157    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911160    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911162    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911165    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911169    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911171    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 06:51:15.913041 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911174    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911177    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911179    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911181    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911186    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911188    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911191    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911194    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911196    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911199    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911201    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911204    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911206    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911209    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911211    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911214    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911217    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911219    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911222    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911224    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 06:51:15.913526 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911226    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911230    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911232    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911235    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911238    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911240    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911243    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911246    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911248    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911251    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911253    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911256    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911259    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911261    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911264    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911268    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911272    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911279    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911282    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 06:51:15.914052 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911284    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911287    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911289    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911292    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911294    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911298    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911300    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911303    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911306    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911308    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911311    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911314    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911316    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911319    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911321    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911324    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911327    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911330    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911333    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911337    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 06:51:15.914511 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911340    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911343    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911753    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911761    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911765    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911768    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911771    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911774    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911777    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911780    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911784    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911787    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911790    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911792    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911795    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911797    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911800    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911802    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911806    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 06:51:15.915025 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911809    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911812    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911815    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911817    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911820    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911823    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911825    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911828    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911830    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911833    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911835    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911838    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911841    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911844    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911847    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911849    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911852    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911855    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911858    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911861    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 06:51:15.915487 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911864    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911866    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911869    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911872    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911878    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911881    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911884    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911886    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911889    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911891    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911895    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911898    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911902    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911906    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911909    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911911    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911914    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911917    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911920    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 06:51:15.915991 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911924    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911927    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911930    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911933    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911936    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911939    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911942    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911945    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911948    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911950    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911953    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911955    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911957    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911960    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911962    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911965    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911967    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911969    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911975    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911978    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 06:51:15.916430 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911980    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911983    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911985    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911988    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911990    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911993    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911995    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.911998    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912000    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912003    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912005    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912008    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912010    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912013    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912015    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912018    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912022    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912025    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912027    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912030    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 06:51:15.916966 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912032    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912035    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912037    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912103    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912135    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912140    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912144    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912148    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912152    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912156    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912160    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912164    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912167    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912170    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912174    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912177    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912180    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912183    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912185    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912188    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912191    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912196    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912199    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 06:51:15.917444 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912202    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912205    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912208    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912211    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912214    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912219    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912222    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912226    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912229    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912232    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912235    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912238    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912242    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912245    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912248    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912251    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912254    2562 flags.go:64] FLAG: --enable-server="true"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912257    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912260    2562 flags.go:64] FLAG: --event-burst="100"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912264    2562 flags.go:64] FLAG: --event-qps="50"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912266    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912270    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912273    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912276    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 06:51:15.918007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912279    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912282    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912285    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912288    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912291    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912293    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912296    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912299    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912302    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912305    2562 flags.go:64] FLAG: --feature-gates=""
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912308    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912312    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912315    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912319    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912322    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912326    2562 flags.go:64] FLAG: --help="false"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912329    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-142-122.ec2.internal"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912332    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912335    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912338    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912341    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912344    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912347    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 06:51:15.918568 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912350    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912353    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912356    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912359    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912361    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912364    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912367    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912370    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912374    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912377    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912379    2562 flags.go:64] FLAG: --lock-file=""
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912382    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912385    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912389    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912394    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912397    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912400    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912403    2562 flags.go:64] FLAG: --logging-format="text"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912405    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912409    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912411    2562 flags.go:64] FLAG: --manifest-url=""
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912414    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912419    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912422    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912427    2562 flags.go:64] FLAG: --max-pods="110"
Sep 09 06:51:15.919227 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912430    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912434    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912437    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912440    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912443    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912446    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912448    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912455    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912458    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912461    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912464    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912467    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912472    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912475    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912478    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912482    2562 flags.go:64] FLAG: --port="10250"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912485    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912488    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e6077bbee7c1466a"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912491    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912494    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912497    2562 flags.go:64] FLAG: --register-node="true"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912500    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912503    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912506    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 09 06:51:15.919847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912509    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912512    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912515    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912518    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912521    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912524    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912530    2562 flags.go:64] FLAG: --runonce="false"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912534    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912537    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912540    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912543    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912546    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912549    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912552    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912555    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912558    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912560    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912563    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912566    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912569    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912572    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912575    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912581    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912583    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912587    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 06:51:15.920423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912591    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912594    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912596    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912599    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912603    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912605    2562 flags.go:64] FLAG: --v="2"
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912609    2562 flags.go:64] FLAG: --version="false"
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912613    2562 flags.go:64] FLAG: --vmodule=""
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912616    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.912619    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912742    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912745    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912749    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912752    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912756    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912759    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912762    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912764    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912767    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912769    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912772    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 06:51:15.921045 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912776    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912780    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912783    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912786    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912789    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912791    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912794    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912796    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912799    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912801    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912804    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912807    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912810    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912812    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912815    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912817    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912820    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912822    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912825    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912828    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 06:51:15.921555 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912830    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912833    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912836    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912838    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912841    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912846    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912848    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912851    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912853    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912856    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912859    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912861    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912864    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912866    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912869    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912871    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912874    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912876    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912879    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 06:51:15.922068 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912881    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912884    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912886    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912889    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912892    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912895    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912899    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912902    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912905    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912907    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912910    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912912    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912915    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912917    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912920    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912922    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912925    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912928    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912932    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912934    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 06:51:15.922518 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912937    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912940    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912943    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912945    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912948    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912950    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912953    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912955    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912958    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912962    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912965    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912968    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912971    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912973    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912976    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912978    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912981    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912983    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912986    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 06:51:15.923069 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912989    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912991    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912994    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912997    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.912999    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.913001    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.913004    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.913006    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.913009    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.913011    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.913581    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.920271    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.920286    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.921153    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 06:51:15.923545 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.921160    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921216    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921221    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921224    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921227    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921230    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921233    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921236    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921239    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921241    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921245    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921247    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921250    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921253    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921256    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921259    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921261    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921263    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921266    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 06:51:15.923898 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921269    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921271    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921274    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921277    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921279    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921282    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921285    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921287    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921290    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921292    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921295    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921299    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921302    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921305    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921307    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921310    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921312    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921315    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921318    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921320    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 06:51:15.924353 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921323    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921325    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921328    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921331    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921333    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921336    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921338    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921342    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921346    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921349    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921352    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921355    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921358    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921360    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921363    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921366    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921369    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921371    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921374    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921377    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 06:51:15.924856 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921379    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921382    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921385    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921388    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921391    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921393    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921396    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921399    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921401    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921404    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921406    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921409    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921411    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921414    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921416    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921419    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921421    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921423    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921426    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921428    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 06:51:15.925328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921431    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921434    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921436    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921440    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921444    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921446    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921449    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921453    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921457    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921461    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921463    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921466    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921469    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921471    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921474    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921477    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921479    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921483    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921485    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 06:51:15.925822 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921488    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921490    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.921495    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921607    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921612    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921615    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921618    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921620    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921623    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921626    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921629    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921632    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921635    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921638    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 06:51:15.926319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921640    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921643    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921646    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921649    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921651    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921654    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921657    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921659    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921662    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921665    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921667    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921670    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921672    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921675    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921678    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921680    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921683    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921686    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921688    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921691    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 06:51:15.926662 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921693    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921708    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921712    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921716    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921720    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921733    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921736    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921738    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921741    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921743    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921746    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921748    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921751    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921753    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921756    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921758    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921761    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921763    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921766    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921769    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 06:51:15.927282 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921772    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921777    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921779    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921782    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921785    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921788    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921790    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921793    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921795    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921798    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921801    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921804    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921806    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921809    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921811    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921814    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921816    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921818    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921821    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921824    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 06:51:15.927786 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921826    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921829    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921831    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921834    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921836    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921839    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921841    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921844    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921846    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921850    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921853    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921856    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921860    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921863    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921866    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921869    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921872    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921875    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921877    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 06:51:15.928271 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921880    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921883    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921885    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921888    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921891    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921894    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921897    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921899    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:15.921902    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.921907    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.922009    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.924659    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.925459    2562 server.go:1019] "Starting client certificate rotation"
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.925557    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 06:51:15.928729 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.925595    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 06:51:15.929182 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.928411    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 06:51:15.932939 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.932918    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 06:51:15.941004 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.940985    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 09 06:51:15.943834 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.943815    2562 log.go:25] "Validated CRI v1 image API"
Sep 09 06:51:15.945178 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.945160    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 06:51:15.947956 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.947931    2562 fs.go:135] Filesystem UUIDs: map[37a178dd-5632-45af-86d8-f554ae1cbc47:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a6bba313-91de-400e-b953-0740b9863388:/dev/nvme0n1p4]
Sep 09 06:51:15.948010 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.947955    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 06:51:15.952044 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.951950    2562 manager.go:216] Machine: {Timestamp:2026-09-09 06:51:15.951488616 +0000 UTC m=+0.290133167 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091495 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2114ea315f67262ab386a75e50a59b SystemUUID:ec2114ea-315f-6726-2ab3-86a75e50a59b BootID:9d864b02-8d41-4ca5-b8b2-3b52c918b3e3 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d0:c1:cf:4e:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d0:c1:cf:4e:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:c5:2f:a1:5a:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 06:51:15.952044 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.952038    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 06:51:15.952158 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.952134    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 06:51:15.954310 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.954295    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 06:51:15.954782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.954755    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 06:51:15.954920 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.954784    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 06:51:15.955521 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.955511    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 06:51:15.955555 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.955523    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 06:51:15.955555 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.955531    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 06:51:15.956142 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.956132    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 06:51:15.957590 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.957580    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 06:51:15.961880 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.961869    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 06:51:15.961993 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.961984    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 06:51:15.962022 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.962014    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 06:51:15.962049 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.962027    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 06:51:15.962076 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.962049    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 06:51:15.962076 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.962062    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 09 06:51:15.962076 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.962074    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 06:51:15.962449 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.962437    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 06:51:15.962506 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.962459    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 06:51:15.965170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.965153    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 06:51:15.967458 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.967445    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 06:51:15.967500 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.967474    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 06:51:15.968124 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968114    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 06:51:15.968155 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968130    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 06:51:15.968155 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968136    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 06:51:15.968155 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968141    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 06:51:15.968155 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968149    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 06:51:15.968155 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968155    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 06:51:15.968279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968168    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 06:51:15.968279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968174    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 06:51:15.968279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968180    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 06:51:15.968279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968186    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 06:51:15.968279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968199    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 06:51:15.968279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968204    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 06:51:15.968279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968228    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 06:51:15.968279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.968233    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 06:51:15.973302 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.973284    2562 server.go:1268] "Started kubelet"
Sep 09 06:51:15.973406 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.973347    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 06:51:15.973406 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.973366    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 06:51:15.973502 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.973426    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 06:51:15.974007 ip-10-0-142-122 systemd[1]: Started Kubernetes Kubelet.
Sep 09 06:51:15.974501 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.974482    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 06:51:15.974604 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.974555    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 06:51:15.976882 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.976815    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 06:51:15.976989 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:15.976966    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 06:51:15.977925 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:15.977902    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 06:51:15.978222 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:15.978118    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 06:51:15.981122 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981107    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 06:51:15.981207 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981123    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 06:51:15.981207 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:15.981169    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:15.981437 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:15.980553    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-122.ec2.internal.18d3946db90db0ae  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-122.ec2.internal,UID:ip-10-0-142-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-122.ec2.internal,},FirstTimestamp:2026-09-09 06:51:15.97324715 +0000 UTC m=+0.311891700,LastTimestamp:2026-09-09 06:51:15.97324715 +0000 UTC m=+0.311891700,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-122.ec2.internal,}"
Sep 09 06:51:15.981804 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981786    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 06:51:15.981804 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981807    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 06:51:15.981954 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981903    2562 factory.go:55] Registering systemd factory
Sep 09 06:51:15.981954 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981918    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 09 06:51:15.981954 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981922    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 06:51:15.981954 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981929    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 06:51:15.981954 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.981932    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 06:51:15.982175 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.982166    2562 factory.go:153] Registering CRI-O factory
Sep 09 06:51:15.982230 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.982177    2562 factory.go:223] Registration of the crio container factory successfully
Sep 09 06:51:15.982283 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.982262    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 06:51:15.982283 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.982280    2562 factory.go:103] Registering Raw factory
Sep 09 06:51:15.982374 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.982300    2562 manager.go:1196] Started watching for new ooms in manager
Sep 09 06:51:15.982718 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.982687    2562 manager.go:349] Starting recovery of all containers
Sep 09 06:51:15.985772 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.985748    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-stz7t"
Sep 09 06:51:15.989730 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:15.989688    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 06:51:15.990179 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:15.990150    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 06:51:15.993107 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.993091    2562 manager.go:354] Recovery completed
Sep 09 06:51:15.995271 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.995250    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-stz7t"
Sep 09 06:51:15.999569 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:15.999556    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 06:51:16.001535 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.001515    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 06:51:16.001612 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.001552    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 06:51:16.001612 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.001567    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 06:51:16.002107 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.002093    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 06:51:16.002168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.002097    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 06:51:16.002168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.002125    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 06:51:16.002168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.002139    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 06:51:16.002746 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.002732    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 06:51:16.002800 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.002756    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 06:51:16.002800 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.002766    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 06:51:16.004091 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.004025    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-122.ec2.internal.18d3946dbabd50d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-122.ec2.internal,UID:ip-10-0-142-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-122.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-122.ec2.internal,},FirstTimestamp:2026-09-09 06:51:16.001534164 +0000 UTC m=+0.340178719,LastTimestamp:2026-09-09 06:51:16.001534164 +0000 UTC m=+0.340178719,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-122.ec2.internal,}"
Sep 09 06:51:16.004177 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.004168    2562 policy_none.go:49] "None policy: Start"
Sep 09 06:51:16.004231 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.004182    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 06:51:16.004231 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.004192    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 06:51:16.006066 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.006056    2562 policy_none.go:47] "Start"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.049267    2562 manager.go:337] "Starting Device Plugin manager"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.049316    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.049330    2562 server.go:87] "Starting device plugin registration server"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.049564    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.049577    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.049652    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.049739    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.049749    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.050240    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 06:51:16.080770 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.050291    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.103725 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.103678    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 06:51:16.104886 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.104865    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 06:51:16.104992 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.104914    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 06:51:16.104992 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.104938    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 06:51:16.104992 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.104982    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 06:51:16.107652 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.107633    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 06:51:16.149821 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.149793    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 06:51:16.150692 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.150674    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 06:51:16.150781 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.150729    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 06:51:16.150781 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.150741    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 06:51:16.150781 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.150766    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.166045 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.165988    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.166045 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.166009    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-122.ec2.internal\": node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.205393 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.205373    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal"]
Sep 09 06:51:16.205482 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.205439    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 06:51:16.206965 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.206951    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 06:51:16.207050 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.206982    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 06:51:16.207050 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.207004    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 06:51:16.208517 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.208502    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 06:51:16.208642 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.208627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.208691 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.208661    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 06:51:16.209218 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.209201    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 06:51:16.209292 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.209228    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 06:51:16.209292 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.209238    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 06:51:16.209292 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.209258    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 06:51:16.209292 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.209240    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 06:51:16.209292 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.209276    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 06:51:16.210487 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.210470    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.210550 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.210501    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 06:51:16.211103 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.211087    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 06:51:16.211174 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.211117    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 06:51:16.211174 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.211129    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 06:51:16.211934 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.211919    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.232291 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.232270    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-122.ec2.internal\" not found" node="ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.236557 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.236541    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-122.ec2.internal\" not found" node="ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.283481 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.283461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc6642f3dfe5b834de843d46287b0217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal\" (UID: \"cc6642f3dfe5b834de843d46287b0217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.283573 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.283488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc6642f3dfe5b834de843d46287b0217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal\" (UID: \"cc6642f3dfe5b834de843d46287b0217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.283573 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.283505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b67763bc95f0a08ece4881f0237f799d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-122.ec2.internal\" (UID: \"b67763bc95f0a08ece4881f0237f799d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.312747 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.312726    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.383870 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.383844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b67763bc95f0a08ece4881f0237f799d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-122.ec2.internal\" (UID: \"b67763bc95f0a08ece4881f0237f799d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.383997 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.383874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc6642f3dfe5b834de843d46287b0217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal\" (UID: \"cc6642f3dfe5b834de843d46287b0217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.383997 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.383906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc6642f3dfe5b834de843d46287b0217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal\" (UID: \"cc6642f3dfe5b834de843d46287b0217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.383997 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.383937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b67763bc95f0a08ece4881f0237f799d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-122.ec2.internal\" (UID: \"b67763bc95f0a08ece4881f0237f799d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.383997 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.383961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc6642f3dfe5b834de843d46287b0217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal\" (UID: \"cc6642f3dfe5b834de843d46287b0217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.383997 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.383937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc6642f3dfe5b834de843d46287b0217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal\" (UID: \"cc6642f3dfe5b834de843d46287b0217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.413612 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.413591    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.514390 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.514331    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.534456 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.534428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.539284 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:16.539263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:16.615248 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.615091    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.715541 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.715518    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.816108 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.816048    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:16.916635 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:16.916602    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:18.044164 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.044149    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 06:51:18.044291 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.044275    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 06:51:18.116551 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.116518    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 06:46:17 +0000 UTC" deadline="2028-06-10 10:28:38.070505112 +0000 UTC"
Sep 09 06:51:18.116551 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.116547    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15363h37m19.953963291s"
Sep 09 06:51:18.135312 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:18.135288    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:18.153298 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.153269    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 06:51:18.220168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.220139    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 06:51:18.236245 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:18.236222    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:18.284328 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:18.284297    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6642f3dfe5b834de843d46287b0217.slice/crio-3cd4046f66c22a33cece793039ceb23b2af946a8c8b3ba7344b4a3d1246785c7 WatchSource:0}: Error finding container 3cd4046f66c22a33cece793039ceb23b2af946a8c8b3ba7344b4a3d1246785c7: Status 404 returned error can't find the container with id 3cd4046f66c22a33cece793039ceb23b2af946a8c8b3ba7344b4a3d1246785c7
Sep 09 06:51:18.288670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.288651    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:51:18.337290 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:18.337268    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-122.ec2.internal\" not found"
Sep 09 06:51:18.431717 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.431698    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 06:51:18.481403 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:18.481370    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67763bc95f0a08ece4881f0237f799d.slice/crio-ba418ffd32b9e8527da28f6d7a990b276564d2dcfc83085a9729b2e8c0463406 WatchSource:0}: Error finding container ba418ffd32b9e8527da28f6d7a990b276564d2dcfc83085a9729b2e8c0463406: Status 404 returned error can't find the container with id ba418ffd32b9e8527da28f6d7a990b276564d2dcfc83085a9729b2e8c0463406
Sep 09 06:51:18.500600 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.500580    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:18.512229 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.512212    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 06:51:18.513020 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.513009    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal"
Sep 09 06:51:18.522561 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.522548    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 06:51:18.944320 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:18.944249    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 06:51:19.082292 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.082025    2562 apiserver.go:52] "Watching apiserver"
Sep 09 06:51:19.088567 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.088542    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 06:51:19.089813 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.089787    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-fct4f","openshift-cluster-node-tuning-operator/tuned-bj269","openshift-dns/node-resolver-cp5xk","openshift-image-registry/node-ca-fjhls","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal","openshift-multus/multus-additional-cni-plugins-72zrg","openshift-multus/multus-f9zfx","openshift-network-operator/iptables-alerter-428wq","kube-system/konnectivity-agent-qzwlw","kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7","openshift-multus/network-metrics-daemon-mfdk5","openshift-network-diagnostics/network-check-target-kjl5t","openshift-ovn-kubernetes/ovnkube-node-lnrr4"]
Sep 09 06:51:19.091622 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.091602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.092864 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.092819    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.094538 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.094340    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 06:51:19.094538 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.094380    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kn26q\""
Sep 09 06:51:19.094691 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.094623    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 06:51:19.094691 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.094670    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 06:51:19.095145 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.094842    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.095145 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.094919    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 06:51:19.095510 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.095491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 06:51:19.095826 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.095709    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 06:51:19.095968 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.095916    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qqkbb\""
Sep 09 06:51:19.096236 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.096219    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 06:51:19.096496 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.096479    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.097136 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.097116    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dwxjt\""
Sep 09 06:51:19.097370 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.097317    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 06:51:19.097370 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.097356    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 06:51:19.097765 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.097746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.098815 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.098794    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 06:51:19.098901 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.098867    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 06:51:19.098966 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.098932    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-psk5w\""
Sep 09 06:51:19.099018 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.098966    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 06:51:19.099213 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.099197    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.101083 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.101064    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 06:51:19.101344 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.101316    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 06:51:19.101410 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.101355    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 06:51:19.101571 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.101544    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 06:51:19.101631 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.101593    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 06:51:19.101771 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.101757    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 06:51:19.101995 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.101971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:19.102088 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.102073    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2hhfc\""
Sep 09 06:51:19.102317 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.102300    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 06:51:19.102456 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.102437    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 06:51:19.102521 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.102501    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 06:51:19.102596 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.102577    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lh8xw\""
Sep 09 06:51:19.103200 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.103182    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:19.103335 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.103247    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:19.108095 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.107737    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 06:51:19.108689 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.108511    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lxk2j\""
Sep 09 06:51:19.109170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.109096    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 06:51:19.110949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.109340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.110949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.109448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.110949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.110756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.110949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.110795    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.110949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.110853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-netns\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.110949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.110885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-systemd-units\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.111492 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.110922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-node-log\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.111492 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111397    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovnkube-script-lib\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.111492 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-daemon-config\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.111492 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.111713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-tmp-dir\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.111713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.111713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111575    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-etc-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.111713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-cni-bin\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.111713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm88n\" (UniqueName: \"kubernetes.io/projected/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-kube-api-access-nm88n\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.111713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-cni-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.111713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-cni-bin\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-etc-kubernetes\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5n6q\" (UniqueName: \"kubernetes.io/projected/40e1d40b-3363-4296-9f1f-873d3d5312aa-kube-api-access-v5n6q\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e27fe24-fa2b-4e20-869b-8c520d98c116-host\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-multus-certs\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111852    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-env-overrides\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovn-node-metrics-cert\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrgt8\" (UniqueName: \"kubernetes.io/projected/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-kube-api-access-vrgt8\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.112033 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.111995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdz8t\" (UniqueName: \"kubernetes.io/projected/c8fe6d86-9c3c-45bb-ba16-713c85718256-kube-api-access-qdz8t\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112088    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-socket-dir-parent\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-system-cni-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-conf-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112365    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db7e0b66-9805-4244-884e-c1822ee70b62-agent-certs\") pod \"konnectivity-agent-qzwlw\" (UID: \"db7e0b66-9805-4244-884e-c1822ee70b62\") " pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:19.112455 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-hosts-file\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mn8k\" (UniqueName: \"kubernetes.io/projected/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-kube-api-access-8mn8k\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.112617    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzsv\" (UniqueName: \"kubernetes.io/projected/9e27fe24-fa2b-4e20-869b-8c520d98c116-kube-api-access-9bzsv\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-run-netns\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-k8s-cni-cncf-io\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.112947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-var-lib-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-cni-netd\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.112994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e1d40b-3363-4296-9f1f-873d3d5312aa-cni-binary-copy\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113095    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-systemd\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-log-socket\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113156    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t7hvw\""
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovnkube-config\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-kubelet\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-slash\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.113362 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-cnibin\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.113822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113385    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 06:51:19.113822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113399    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e27fe24-fa2b-4e20-869b-8c520d98c116-serviceca\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.113822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-ovn\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.113822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-hostroot\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.113822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113608    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 06:51:19.113822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db7e0b66-9805-4244-884e-c1822ee70b62-konnectivity-ca\") pod \"konnectivity-agent-qzwlw\" (UID: \"db7e0b66-9805-4244-884e-c1822ee70b62\") " pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:19.113822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-cni-multus\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.113822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-kubelet\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.114193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-device-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.114193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113845    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 06:51:19.114193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-system-cni-dir\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.114193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.113977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cnibin\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.115389 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.115370    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:19.115478 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.115455    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:19.115766 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.115736    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 06:51:19.115861 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.115846    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pfhz4\""
Sep 09 06:51:19.118308 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.118287    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 06:51:19.141070 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.140343    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2h4k"
Sep 09 06:51:19.149410 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.149375    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2h4k"
Sep 09 06:51:19.201752 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.201684    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 06:51:19.215042 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.215207 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-var-lib-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215207 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-cni-netd\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215317 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215255    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-etc-selinux\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.215317 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-cni-netd\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215392 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-var-lib-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215392 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7kq\" (UniqueName: \"kubernetes.io/projected/ee1df979-5286-453c-bb5c-68f35263ef3d-kube-api-access-7z7kq\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.215392 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-var-lib-kubelet\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.215392 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-tuned\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.215560 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:19.215560 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e1d40b-3363-4296-9f1f-873d3d5312aa-cni-binary-copy\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.215560 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.215560 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-systemd\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215560 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215544    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-log-socket\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215776 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovnkube-config\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215776 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-log-socket\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215776 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-dbus\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.215776 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.215949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45c7\" (UniqueName: \"kubernetes.io/projected/20f229be-0318-40fb-aa2a-62e0df181580-kube-api-access-j45c7\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:19.215949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-kubelet\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-slash\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.215949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-cnibin\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.215980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e27fe24-fa2b-4e20-869b-8c520d98c116-serviceca\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.216193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-ovn\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.216193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-systemd\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.216193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-hostroot\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-hostroot\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db7e0b66-9805-4244-884e-c1822ee70b62-konnectivity-ca\") pod \"konnectivity-agent-qzwlw\" (UID: \"db7e0b66-9805-4244-884e-c1822ee70b62\") " pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:19.216193 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-cni-multus\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-kubelet\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216211    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-device-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-system-cni-dir\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216277    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-kubelet\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cnibin\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-device-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-cni-multus\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-system-cni-dir\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cnibin\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-cnibin\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-modprobe-d\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-slash\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-netns\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.216534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-systemd-units\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-kubelet\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-systemd-units\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-ovn\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-node-log\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216649    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-netns\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovnkube-script-lib\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovnkube-config\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216708    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db7e0b66-9805-4244-884e-c1822ee70b62-konnectivity-ca\") pod \"konnectivity-agent-qzwlw\" (UID: \"db7e0b66-9805-4244-884e-c1822ee70b62\") " pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysctl-conf\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-systemd\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-run-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-sys\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-tmp\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216877    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-node-log\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-daemon-config\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.217336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.216991    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-tmp-dir\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217032    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e27fe24-fa2b-4e20-869b-8c520d98c116-serviceca\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-etc-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40e1d40b-3363-4296-9f1f-873d3d5312aa-cni-binary-copy\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217189    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovnkube-script-lib\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-etc-openvswitch\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-cni-bin\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-cni-bin\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-run\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm88n\" (UniqueName: \"kubernetes.io/projected/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-kube-api-access-nm88n\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-cni-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-daemon-config\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-cni-bin\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-tmp-dir\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-etc-kubernetes\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-etc-kubernetes\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.218196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5n6q\" (UniqueName: \"kubernetes.io/projected/40e1d40b-3363-4296-9f1f-873d3d5312aa-kube-api-access-v5n6q\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217527    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e27fe24-fa2b-4e20-869b-8c520d98c116-host\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217456    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-var-lib-cni-bin\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee1df979-5286-453c-bb5c-68f35263ef3d-host-slash\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e27fe24-fa2b-4e20-869b-8c520d98c116-host\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217649    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-cni-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-kubernetes\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-multus-certs\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-env-overrides\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovn-node-metrics-cert\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-multus-certs\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysconfig\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-lib-modules\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgt8\" (UniqueName: \"kubernetes.io/projected/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-kube-api-access-vrgt8\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217916    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdz8t\" (UniqueName: \"kubernetes.io/projected/c8fe6d86-9c3c-45bb-ba16-713c85718256-kube-api-access-qdz8t\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.219035 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ee1df979-5286-453c-bb5c-68f35263ef3d-iptables-alerter-script\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.217990    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-socket-dir-parent\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-kubelet-config\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-system-cni-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-conf-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8fe6d86-9c3c-45bb-ba16-713c85718256-env-overrides\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218135    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218234    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-socket-dir-parent\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-registration-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db7e0b66-9805-4244-884e-c1822ee70b62-agent-certs\") pod \"konnectivity-agent-qzwlw\" (UID: \"db7e0b66-9805-4244-884e-c1822ee70b62\") " pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-host\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:19.219838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-system-cni-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-hosts-file\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mn8k\" (UniqueName: \"kubernetes.io/projected/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-kube-api-access-8mn8k\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzsv\" (UniqueName: \"kubernetes.io/projected/9e27fe24-fa2b-4e20-869b-8c520d98c116-kube-api-access-9bzsv\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-run-netns\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysctl-d\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-socket-dir\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-k8s-cni-cncf-io\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zm5s\" (UniqueName: \"kubernetes.io/projected/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-kube-api-access-5zm5s\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-host-run-k8s-cni-cncf-io\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218708    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-hosts-file\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218481    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-sys-fs\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.218510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40e1d40b-3363-4296-9f1f-873d3d5312aa-multus-conf-dir\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.219207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.220595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.219258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fe6d86-9c3c-45bb-ba16-713c85718256-host-run-netns\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.222853 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.222821    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8fe6d86-9c3c-45bb-ba16-713c85718256-ovn-node-metrics-cert\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.222949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.222911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db7e0b66-9805-4244-884e-c1822ee70b62-agent-certs\") pod \"konnectivity-agent-qzwlw\" (UID: \"db7e0b66-9805-4244-884e-c1822ee70b62\") " pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:19.228168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.228119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm88n\" (UniqueName: \"kubernetes.io/projected/8fcfdca1-f156-4eb3-be22-9a8d6be237a8-kube-api-access-nm88n\") pod \"aws-ebs-csi-driver-node-vwfz7\" (UID: \"8fcfdca1-f156-4eb3-be22-9a8d6be237a8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.229806 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.229787    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdz8t\" (UniqueName: \"kubernetes.io/projected/c8fe6d86-9c3c-45bb-ba16-713c85718256-kube-api-access-qdz8t\") pod \"ovnkube-node-lnrr4\" (UID: \"c8fe6d86-9c3c-45bb-ba16-713c85718256\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.230398 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.230378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5n6q\" (UniqueName: \"kubernetes.io/projected/40e1d40b-3363-4296-9f1f-873d3d5312aa-kube-api-access-v5n6q\") pod \"multus-f9zfx\" (UID: \"40e1d40b-3363-4296-9f1f-873d3d5312aa\") " pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.230486 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.230409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgt8\" (UniqueName: \"kubernetes.io/projected/3cd0276a-00b2-49c9-86a9-4bd63ccf9457-kube-api-access-vrgt8\") pod \"node-resolver-cp5xk\" (UID: \"3cd0276a-00b2-49c9-86a9-4bd63ccf9457\") " pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.230730 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.230688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal" event={"ID":"b67763bc95f0a08ece4881f0237f799d","Type":"ContainerStarted","Data":"ba418ffd32b9e8527da28f6d7a990b276564d2dcfc83085a9729b2e8c0463406"}
Sep 09 06:51:19.230820 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.230802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzsv\" (UniqueName: \"kubernetes.io/projected/9e27fe24-fa2b-4e20-869b-8c520d98c116-kube-api-access-9bzsv\") pod \"node-ca-fjhls\" (UID: \"9e27fe24-fa2b-4e20-869b-8c520d98c116\") " pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.232910 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.232855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mn8k\" (UniqueName: \"kubernetes.io/projected/24c3a8e5-a25d-4de8-a74d-bb19e329f6f1-kube-api-access-8mn8k\") pod \"multus-additional-cni-plugins-72zrg\" (UID: \"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1\") " pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.233752 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.233727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal" event={"ID":"cc6642f3dfe5b834de843d46287b0217","Type":"ContainerStarted","Data":"3cd4046f66c22a33cece793039ceb23b2af946a8c8b3ba7344b4a3d1246785c7"}
Sep 09 06:51:19.319714 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-modprobe-d\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.319887 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysctl-conf\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.319887 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-systemd\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.319887 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-sys\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.319887 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-tmp\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.319887 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-run\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.319887 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee1df979-5286-453c-bb5c-68f35263ef3d-host-slash\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.319887 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-modprobe-d\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.319887 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319875    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-systemd\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysctl-conf\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-kubernetes\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-run\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-kubernetes\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee1df979-5286-453c-bb5c-68f35263ef3d-host-slash\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysconfig\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.319994    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-sys\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysconfig\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-lib-modules\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ee1df979-5286-453c-bb5c-68f35263ef3d-iptables-alerter-script\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-kubelet-config\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-host\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320229    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysctl-d\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zm5s\" (UniqueName: \"kubernetes.io/projected/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-kube-api-access-5zm5s\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7kq\" (UniqueName: \"kubernetes.io/projected/ee1df979-5286-453c-bb5c-68f35263ef3d-kube-api-access-7z7kq\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-host\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320279    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-lib-modules\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-var-lib-kubelet\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320477    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-sysctl-d\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.320528    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:19.320566 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-var-lib-kubelet\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-tuned\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-kubelet-config\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.320625    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs podName:20f229be-0318-40fb-aa2a-62e0df181580 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:19.820583268 +0000 UTC m=+3.040824370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs") pod "network-metrics-daemon-mfdk5" (UID: "20f229be-0318-40fb-aa2a-62e0df181580") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-dbus\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45c7\" (UniqueName: \"kubernetes.io/projected/20f229be-0318-40fb-aa2a-62e0df181580-kube-api-access-j45c7\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320850    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ee1df979-5286-453c-bb5c-68f35263ef3d-iptables-alerter-script\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.320886    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-dbus\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.320942    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:19.321163 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.320992    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret podName:2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d nodeName:}" failed. No retries permitted until 2026-09-09 06:51:19.82097499 +0000 UTC m=+3.041216102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret") pod "global-pull-secret-syncer-fct4f" (UID: "2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:19.322464 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.322438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-tmp\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.323499 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.323482    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-etc-tuned\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.327567 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.327543    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 06:51:19.327687 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.327570    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 06:51:19.327687 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.327586    2562 projected.go:196] Error preparing data for projected volume kube-api-access-l2k9s for pod openshift-network-diagnostics/network-check-target-kjl5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:19.327687 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.327649    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s podName:9552e261-a17b-4376-99b3-5ff9f8f87b0e nodeName:}" failed. No retries permitted until 2026-09-09 06:51:19.827632485 +0000 UTC m=+3.047873578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l2k9s" (UniqueName: "kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s") pod "network-check-target-kjl5t" (UID: "9552e261-a17b-4376-99b3-5ff9f8f87b0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:19.329588 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.329566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zm5s\" (UniqueName: \"kubernetes.io/projected/2490c495-e01b-4e49-9e16-7fb3ed27a2f1-kube-api-access-5zm5s\") pod \"tuned-bj269\" (UID: \"2490c495-e01b-4e49-9e16-7fb3ed27a2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.330210 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.329965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7kq\" (UniqueName: \"kubernetes.io/projected/ee1df979-5286-453c-bb5c-68f35263ef3d-kube-api-access-7z7kq\") pod \"iptables-alerter-428wq\" (UID: \"ee1df979-5286-453c-bb5c-68f35263ef3d\") " pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.330423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.330406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45c7\" (UniqueName: \"kubernetes.io/projected/20f229be-0318-40fb-aa2a-62e0df181580-kube-api-access-j45c7\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:19.415076 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.415038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f9zfx"
Sep 09 06:51:19.420836 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.420816    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7"
Sep 09 06:51:19.430398 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.430379    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cp5xk"
Sep 09 06:51:19.436083 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.436061    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fjhls"
Sep 09 06:51:19.445690 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.445672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:19.455376 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.455328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72zrg"
Sep 09 06:51:19.460870 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.460850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:19.470670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.470642    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-428wq"
Sep 09 06:51:19.478226 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.478208    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bj269"
Sep 09 06:51:19.824451 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.824370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:19.824451 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.824441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:19.824664 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.824529    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:19.824664 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.824553    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:19.824664 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.824604    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret podName:2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d nodeName:}" failed. No retries permitted until 2026-09-09 06:51:20.824584459 +0000 UTC m=+4.044825551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret") pod "global-pull-secret-syncer-fct4f" (UID: "2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:19.824664 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.824620    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs podName:20f229be-0318-40fb-aa2a-62e0df181580 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:20.824613115 +0000 UTC m=+4.044854206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs") pod "network-metrics-daemon-mfdk5" (UID: "20f229be-0318-40fb-aa2a-62e0df181580") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:19.925024 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:19.924994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:19.925210 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.925192    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 06:51:19.925264 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.925217    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 06:51:19.925264 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.925229    2562 projected.go:196] Error preparing data for projected volume kube-api-access-l2k9s for pod openshift-network-diagnostics/network-check-target-kjl5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:19.925363 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:19.925288    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s podName:9552e261-a17b-4376-99b3-5ff9f8f87b0e nodeName:}" failed. No retries permitted until 2026-09-09 06:51:20.925272934 +0000 UTC m=+4.145514043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2k9s" (UniqueName: "kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s") pod "network-check-target-kjl5t" (UID: "9552e261-a17b-4376-99b3-5ff9f8f87b0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:20.152043 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:20.151968    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 06:46:19 +0000 UTC" deadline="2028-04-02 12:00:23.015548784 +0000 UTC"
Sep 09 06:51:20.152043 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:20.151998    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13709h9m2.863554043s"
Sep 09 06:51:20.272895 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.272810    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c3a8e5_a25d_4de8_a74d_bb19e329f6f1.slice/crio-58de4bdfcd9a22b72ad527ef64afc3551cf94625d111e29e31f6f749a02b066a WatchSource:0}: Error finding container 58de4bdfcd9a22b72ad527ef64afc3551cf94625d111e29e31f6f749a02b066a: Status 404 returned error can't find the container with id 58de4bdfcd9a22b72ad527ef64afc3551cf94625d111e29e31f6f749a02b066a
Sep 09 06:51:20.286616 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.286596    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fe6d86_9c3c_45bb_ba16_713c85718256.slice/crio-71fc1f26b175a667ab79c35c5032698173d53e360d20737a6fdb4faad73b347d WatchSource:0}: Error finding container 71fc1f26b175a667ab79c35c5032698173d53e360d20737a6fdb4faad73b347d: Status 404 returned error can't find the container with id 71fc1f26b175a667ab79c35c5032698173d53e360d20737a6fdb4faad73b347d
Sep 09 06:51:20.288124 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.288092    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1df979_5286_453c_bb5c_68f35263ef3d.slice/crio-04c74c2c253909d696024404c65eb5a718d2698bd5b66b182c1fcbe8437d5dde WatchSource:0}: Error finding container 04c74c2c253909d696024404c65eb5a718d2698bd5b66b182c1fcbe8437d5dde: Status 404 returned error can't find the container with id 04c74c2c253909d696024404c65eb5a718d2698bd5b66b182c1fcbe8437d5dde
Sep 09 06:51:20.289382 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.289361    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd0276a_00b2_49c9_86a9_4bd63ccf9457.slice/crio-44feb0582ab4aeed43eb5cbaefe426d93c01f455af45d55f7b01f3b95abec3ff WatchSource:0}: Error finding container 44feb0582ab4aeed43eb5cbaefe426d93c01f455af45d55f7b01f3b95abec3ff: Status 404 returned error can't find the container with id 44feb0582ab4aeed43eb5cbaefe426d93c01f455af45d55f7b01f3b95abec3ff
Sep 09 06:51:20.290749 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.290654    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcfdca1_f156_4eb3_be22_9a8d6be237a8.slice/crio-21eefa79c3df4a233e635850196547d9f77767fda3840a3cbcedf9674721aadd WatchSource:0}: Error finding container 21eefa79c3df4a233e635850196547d9f77767fda3840a3cbcedf9674721aadd: Status 404 returned error can't find the container with id 21eefa79c3df4a233e635850196547d9f77767fda3840a3cbcedf9674721aadd
Sep 09 06:51:20.291339 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.291309    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e1d40b_3363_4296_9f1f_873d3d5312aa.slice/crio-b93b9e49767c17e83c23c2bdf364e896f56231cba0794bad346aee0785ea65ec WatchSource:0}: Error finding container b93b9e49767c17e83c23c2bdf364e896f56231cba0794bad346aee0785ea65ec: Status 404 returned error can't find the container with id b93b9e49767c17e83c23c2bdf364e896f56231cba0794bad346aee0785ea65ec
Sep 09 06:51:20.292220 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.292196    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e27fe24_fa2b_4e20_869b_8c520d98c116.slice/crio-344cb2b3253876c69d3527c333b826b4b10655a0b616eb4050a8d890dac0b01c WatchSource:0}: Error finding container 344cb2b3253876c69d3527c333b826b4b10655a0b616eb4050a8d890dac0b01c: Status 404 returned error can't find the container with id 344cb2b3253876c69d3527c333b826b4b10655a0b616eb4050a8d890dac0b01c
Sep 09 06:51:20.293719 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.293696    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2490c495_e01b_4e49_9e16_7fb3ed27a2f1.slice/crio-6931ffc77a885ebacfb6f0877cec70b22a6936ac4576f940f06864b346e815b8 WatchSource:0}: Error finding container 6931ffc77a885ebacfb6f0877cec70b22a6936ac4576f940f06864b346e815b8: Status 404 returned error can't find the container with id 6931ffc77a885ebacfb6f0877cec70b22a6936ac4576f940f06864b346e815b8
Sep 09 06:51:20.295870 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:20.295837    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7e0b66_9805_4244_884e_c1822ee70b62.slice/crio-e64a4069cdd97243e218ac222084848675b263151eaedcb83c96225e865f1cf5 WatchSource:0}: Error finding container e64a4069cdd97243e218ac222084848675b263151eaedcb83c96225e865f1cf5: Status 404 returned error can't find the container with id e64a4069cdd97243e218ac222084848675b263151eaedcb83c96225e865f1cf5
Sep 09 06:51:20.833076 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:20.832363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:20.833076 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:20.832493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:20.833076 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:20.832643    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:20.833076 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:20.832704    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret podName:2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d nodeName:}" failed. No retries permitted until 2026-09-09 06:51:22.832686272 +0000 UTC m=+6.052927368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret") pod "global-pull-secret-syncer-fct4f" (UID: "2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:20.833076 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:20.833024    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:20.833076 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:20.833082    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs podName:20f229be-0318-40fb-aa2a-62e0df181580 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:22.833065733 +0000 UTC m=+6.053306830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs") pod "network-metrics-daemon-mfdk5" (UID: "20f229be-0318-40fb-aa2a-62e0df181580") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:20.933422 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:20.932778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:20.933422 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:20.932984    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 06:51:20.933422 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:20.933002    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 06:51:20.933422 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:20.933015    2562 projected.go:196] Error preparing data for projected volume kube-api-access-l2k9s for pod openshift-network-diagnostics/network-check-target-kjl5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:20.933422 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:20.933069    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s podName:9552e261-a17b-4376-99b3-5ff9f8f87b0e nodeName:}" failed. No retries permitted until 2026-09-09 06:51:22.933053166 +0000 UTC m=+6.153294262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2k9s" (UniqueName: "kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s") pod "network-check-target-kjl5t" (UID: "9552e261-a17b-4376-99b3-5ff9f8f87b0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:21.152498 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.152388    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 06:46:19 +0000 UTC" deadline="2028-04-14 22:24:10.299537141 +0000 UTC"
Sep 09 06:51:21.152498 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.152450    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14007h32m49.14709091s"
Sep 09 06:51:21.224946 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.224218    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:21.224946 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.224239    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:21.224946 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.224287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:21.224946 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:21.224564    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:21.224946 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:21.224690    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:21.224946 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:21.224774    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:21.253003 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.252970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f9zfx" event={"ID":"40e1d40b-3363-4296-9f1f-873d3d5312aa","Type":"ContainerStarted","Data":"b93b9e49767c17e83c23c2bdf364e896f56231cba0794bad346aee0785ea65ec"}
Sep 09 06:51:21.264626 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.264596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7" event={"ID":"8fcfdca1-f156-4eb3-be22-9a8d6be237a8","Type":"ContainerStarted","Data":"21eefa79c3df4a233e635850196547d9f77767fda3840a3cbcedf9674721aadd"}
Sep 09 06:51:21.279292 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.279265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72zrg" event={"ID":"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1","Type":"ContainerStarted","Data":"58de4bdfcd9a22b72ad527ef64afc3551cf94625d111e29e31f6f749a02b066a"}
Sep 09 06:51:21.288541 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.288513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bj269" event={"ID":"2490c495-e01b-4e49-9e16-7fb3ed27a2f1","Type":"ContainerStarted","Data":"6931ffc77a885ebacfb6f0877cec70b22a6936ac4576f940f06864b346e815b8"}
Sep 09 06:51:21.295184 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.295092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cp5xk" event={"ID":"3cd0276a-00b2-49c9-86a9-4bd63ccf9457","Type":"ContainerStarted","Data":"44feb0582ab4aeed43eb5cbaefe426d93c01f455af45d55f7b01f3b95abec3ff"}
Sep 09 06:51:21.299275 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.299248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-428wq" event={"ID":"ee1df979-5286-453c-bb5c-68f35263ef3d","Type":"ContainerStarted","Data":"04c74c2c253909d696024404c65eb5a718d2698bd5b66b182c1fcbe8437d5dde"}
Sep 09 06:51:21.314334 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.314302    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"71fc1f26b175a667ab79c35c5032698173d53e360d20737a6fdb4faad73b347d"}
Sep 09 06:51:21.357160 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.357129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal" event={"ID":"b67763bc95f0a08ece4881f0237f799d","Type":"ContainerStarted","Data":"5d2443e04a1fc40253f88b121ce87fff8134d710f27fadbd924916e0d3a23cbc"}
Sep 09 06:51:21.379140 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.379093    2562 generic.go:356] "Generic (PLEG): container finished" podID="cc6642f3dfe5b834de843d46287b0217" containerID="46782f5346fcd232b75e06c442998f8b710b9437727e8309a28484f91b05c0dd" exitCode=0
Sep 09 06:51:21.379268 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.379188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal" event={"ID":"cc6642f3dfe5b834de843d46287b0217","Type":"ContainerDied","Data":"46782f5346fcd232b75e06c442998f8b710b9437727e8309a28484f91b05c0dd"}
Sep 09 06:51:21.396875 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.396826    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-122.ec2.internal" podStartSLOduration=3.39680764 podStartE2EDuration="3.39680764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:21.37315551 +0000 UTC m=+4.593396624" watchObservedRunningTime="2026-09-09 06:51:21.39680764 +0000 UTC m=+4.617048755"
Sep 09 06:51:21.398003 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.397981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzwlw" event={"ID":"db7e0b66-9805-4244-884e-c1822ee70b62","Type":"ContainerStarted","Data":"e64a4069cdd97243e218ac222084848675b263151eaedcb83c96225e865f1cf5"}
Sep 09 06:51:21.416923 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:21.416860    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fjhls" event={"ID":"9e27fe24-fa2b-4e20-869b-8c520d98c116","Type":"ContainerStarted","Data":"344cb2b3253876c69d3527c333b826b4b10655a0b616eb4050a8d890dac0b01c"}
Sep 09 06:51:22.439684 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:22.439641    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal" event={"ID":"cc6642f3dfe5b834de843d46287b0217","Type":"ContainerStarted","Data":"dbf7adf5ec2f6d04ea2329710a7ed3f0ed391a5c26985306533c7355ab8bfc47"}
Sep 09 06:51:22.849634 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:22.849548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:22.849634 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:22.849618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:22.849850 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:22.849774    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:22.849850 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:22.849832    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret podName:2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d nodeName:}" failed. No retries permitted until 2026-09-09 06:51:26.849814256 +0000 UTC m=+10.070055352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret") pod "global-pull-secret-syncer-fct4f" (UID: "2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:22.850471 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:22.850219    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:22.850471 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:22.850273    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs podName:20f229be-0318-40fb-aa2a-62e0df181580 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:26.850258113 +0000 UTC m=+10.070499210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs") pod "network-metrics-daemon-mfdk5" (UID: "20f229be-0318-40fb-aa2a-62e0df181580") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:22.950602 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:22.950566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:22.950806 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:22.950788    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 06:51:22.950877 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:22.950815    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 06:51:22.950877 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:22.950829    2562 projected.go:196] Error preparing data for projected volume kube-api-access-l2k9s for pod openshift-network-diagnostics/network-check-target-kjl5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:22.950978 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:22.950886    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s podName:9552e261-a17b-4376-99b3-5ff9f8f87b0e nodeName:}" failed. No retries permitted until 2026-09-09 06:51:26.950865786 +0000 UTC m=+10.171106903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2k9s" (UniqueName: "kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s") pod "network-check-target-kjl5t" (UID: "9552e261-a17b-4376-99b3-5ff9f8f87b0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:23.233310 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:23.233226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:23.233478 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:23.233421    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:23.233941 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:23.233916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:23.234092 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:23.234059    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:23.234224 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:23.234208    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:23.234368 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:23.234343    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:25.224052 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:25.223933    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:25.224570 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:25.224083    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:25.224570 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:25.224467    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:25.224741 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:25.224568    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:25.224741 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:25.224689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:25.224845 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:25.224760    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:26.883134 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:26.883062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:26.883579 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:26.883141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:26.883579 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:26.883261    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:26.883579 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:26.883324    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret podName:2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d nodeName:}" failed. No retries permitted until 2026-09-09 06:51:34.883307352 +0000 UTC m=+18.103548459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret") pod "global-pull-secret-syncer-fct4f" (UID: "2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:26.883754 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:26.883709    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:26.883805 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:26.883762    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs podName:20f229be-0318-40fb-aa2a-62e0df181580 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:34.883745226 +0000 UTC m=+18.103986331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs") pod "network-metrics-daemon-mfdk5" (UID: "20f229be-0318-40fb-aa2a-62e0df181580") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:26.984257 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:26.984221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:26.984446 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:26.984426    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 06:51:26.984535 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:26.984454    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 06:51:26.984535 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:26.984472    2562 projected.go:196] Error preparing data for projected volume kube-api-access-l2k9s for pod openshift-network-diagnostics/network-check-target-kjl5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:26.984535 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:26.984526    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s podName:9552e261-a17b-4376-99b3-5ff9f8f87b0e nodeName:}" failed. No retries permitted until 2026-09-09 06:51:34.984508772 +0000 UTC m=+18.204749866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2k9s" (UniqueName: "kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s") pod "network-check-target-kjl5t" (UID: "9552e261-a17b-4376-99b3-5ff9f8f87b0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:27.224293 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:27.224222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:27.225335 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:27.225313    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:27.225986 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:27.225458    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:27.225986 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:27.225805    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:27.225986 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:27.225857    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:27.225986 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:27.225951    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:29.225329 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:29.224560    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:29.225329 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:29.224680    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:29.225329 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:29.225177    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:29.225329 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:29.225285    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:29.226191 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:29.226021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:29.226191 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:29.226146    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:31.224268 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:31.224230    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:31.224703 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:31.224377    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:31.224703 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:31.224230    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:31.224900 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:31.224874    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:31.225064 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:31.225022    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:31.227737 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:31.225193    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:33.227031 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:33.227003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:33.227483 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:33.227040    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:33.227483 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:33.227133    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:33.227483 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:33.227162    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:33.227483 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:33.227229    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:33.227483 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:33.227317    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:34.951215 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:34.951179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:34.951697 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:34.951232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:34.951697 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:34.951343    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:34.951697 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:34.951360    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:34.951697 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:34.951415    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs podName:20f229be-0318-40fb-aa2a-62e0df181580 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:50.951398389 +0000 UTC m=+34.171639480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs") pod "network-metrics-daemon-mfdk5" (UID: "20f229be-0318-40fb-aa2a-62e0df181580") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:34.951697 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:34.951431    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret podName:2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d nodeName:}" failed. No retries permitted until 2026-09-09 06:51:50.951424693 +0000 UTC m=+34.171665787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret") pod "global-pull-secret-syncer-fct4f" (UID: "2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:35.051543 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:35.051493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:35.051719 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:35.051622    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 06:51:35.051719 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:35.051639    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 06:51:35.051719 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:35.051649    2562 projected.go:196] Error preparing data for projected volume kube-api-access-l2k9s for pod openshift-network-diagnostics/network-check-target-kjl5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:35.051719 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:35.051702    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s podName:9552e261-a17b-4376-99b3-5ff9f8f87b0e nodeName:}" failed. No retries permitted until 2026-09-09 06:51:51.051688972 +0000 UTC m=+34.271930064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2k9s" (UniqueName: "kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s") pod "network-check-target-kjl5t" (UID: "9552e261-a17b-4376-99b3-5ff9f8f87b0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:35.227654 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:35.227584    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:35.227792 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:35.227695    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:35.227792 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:35.227705    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:35.227792 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:35.227719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:35.227927 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:35.227792    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:35.227927 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:35.227871    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:37.225186 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:37.225149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:37.225666 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:37.225253    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:37.225666 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:37.225266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:37.225666 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:37.225301    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:37.225666 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:37.225400    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:37.225666 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:37.225502    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:38.470674 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.470458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7" event={"ID":"8fcfdca1-f156-4eb3-be22-9a8d6be237a8","Type":"ContainerStarted","Data":"fb20f799293d524797bab33bd7441b824afbb98d3ddde23e90a255c7d5fa0dc1"}
Sep 09 06:51:38.474395 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.474371    2562 generic.go:356] "Generic (PLEG): container finished" podID="24c3a8e5-a25d-4de8-a74d-bb19e329f6f1" containerID="ddaa9ebd30b8af75fe462081015e26dd40c1b9cf8a4bb5c8df9efd2900e80b62" exitCode=0
Sep 09 06:51:38.474485 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.474439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72zrg" event={"ID":"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1","Type":"ContainerDied","Data":"ddaa9ebd30b8af75fe462081015e26dd40c1b9cf8a4bb5c8df9efd2900e80b62"}
Sep 09 06:51:38.475858 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.475840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bj269" event={"ID":"2490c495-e01b-4e49-9e16-7fb3ed27a2f1","Type":"ContainerStarted","Data":"f645f4a274ac9d3941ec7281745d49f6539f3ae62405c78f8186a9065a32730a"}
Sep 09 06:51:38.477285 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.477262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cp5xk" event={"ID":"3cd0276a-00b2-49c9-86a9-4bd63ccf9457","Type":"ContainerStarted","Data":"2bd993f446d28f99183009fd2771913d42c8761d4b05ddc7fa249788abe77237"}
Sep 09 06:51:38.479079 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.479063    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 06:51:38.479421 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.479401    2562 generic.go:356] "Generic (PLEG): container finished" podID="c8fe6d86-9c3c-45bb-ba16-713c85718256" containerID="c520abf08089debcc9f2aa88cb9f791e4632e172634360f98354b8a2227e4ee6" exitCode=1
Sep 09 06:51:38.479515 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.479472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"5ba897536aae40c1b9c53c7ceaed938e8b4814c5c8a3864417bec2b54e3a9e96"}
Sep 09 06:51:38.479556 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.479546    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerDied","Data":"c520abf08089debcc9f2aa88cb9f791e4632e172634360f98354b8a2227e4ee6"}
Sep 09 06:51:38.479621 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.479599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"9b3d7b01a689e58069b8e69e7278de18c2df08055db6f65cf700813f12dcd60f"}
Sep 09 06:51:38.481233 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.481210    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzwlw" event={"ID":"db7e0b66-9805-4244-884e-c1822ee70b62","Type":"ContainerStarted","Data":"ee071d5b13060ce00fa12f66aea60538b5ae9abc81cbdbbaf8bf434495efb049"}
Sep 09 06:51:38.482604 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.482583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fjhls" event={"ID":"9e27fe24-fa2b-4e20-869b-8c520d98c116","Type":"ContainerStarted","Data":"aeae417dbfb948c19d6b83dd3c78e42b450dd9275c3716f75b40652ea6949270"}
Sep 09 06:51:38.496218 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.496180    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-122.ec2.internal" podStartSLOduration=20.496168288 podStartE2EDuration="20.496168288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:22.456294556 +0000 UTC m=+5.676535670" watchObservedRunningTime="2026-09-09 06:51:38.496168288 +0000 UTC m=+21.716409400"
Sep 09 06:51:38.510788 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.510741    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cp5xk" podStartSLOduration=3.801649439 podStartE2EDuration="21.51072377s" totalImagesPullingTime="17.709074331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:38.510020166 +0000 UTC m=+21.730261280" watchObservedRunningTime="2026-09-09 06:51:38.51072377 +0000 UTC m=+21.730964885"
Sep 09 06:51:38.544714 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.544670    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bj269" podStartSLOduration=3.837231877 podStartE2EDuration="21.544654683s" totalImagesPullingTime="17.707422806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:38.544269702 +0000 UTC m=+21.764510829" watchObservedRunningTime="2026-09-09 06:51:38.544654683 +0000 UTC m=+21.764895795"
Sep 09 06:51:38.544836 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.544793    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fjhls" podStartSLOduration=3.835638429 podStartE2EDuration="21.544785428s" totalImagesPullingTime="17.709146999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:38.527706217 +0000 UTC m=+21.747947330" watchObservedRunningTime="2026-09-09 06:51:38.544785428 +0000 UTC m=+21.765026603"
Sep 09 06:51:38.563843 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.563572    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qzwlw" podStartSLOduration=7.900189425 podStartE2EDuration="21.563551571s" totalImagesPullingTime="13.663362146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:38.563210834 +0000 UTC m=+21.783451948" watchObservedRunningTime="2026-09-09 06:51:38.563551571 +0000 UTC m=+21.783792688"
Sep 09 06:51:38.601197 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.601130    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:38.601662 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:38.601647    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:39.224628 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.224597    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:39.224628 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.224627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:39.224821 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.224600    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:39.224821 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:39.224695    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:39.224821 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:39.224793    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:39.224932 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:39.224863    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:39.468279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.468250    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 06:51:39.486325 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.486295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f9zfx" event={"ID":"40e1d40b-3363-4296-9f1f-873d3d5312aa","Type":"ContainerStarted","Data":"0b429ab432501e4ad09c2ece91cb2397059e545967b23049add1c5e2841cb42a"}
Sep 09 06:51:39.489123 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.489087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7" event={"ID":"8fcfdca1-f156-4eb3-be22-9a8d6be237a8","Type":"ContainerStarted","Data":"db0135420774747e021d97cbfbaab2865f67aae6804ffd7b2d45a0ec76e295a9"}
Sep 09 06:51:39.492581 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.492553    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 06:51:39.493036 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.492952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"1e5d6436b2b5a6bc64a08110d469f09a72dc0fd7f8465683579ddd508b21d477"}
Sep 09 06:51:39.493134 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.493032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"ecf2a9d80b127adf74531c84ddb14ce0545ea2c7a411c30baadc6b011a7c8022"}
Sep 09 06:51:39.493134 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.493094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"ea206caf6a7996bd2f7b9f2fe835b16b1a85c2a7fe414e12b1676fb160988622"}
Sep 09 06:51:39.493394 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.493371    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:39.493845 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.493818    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzwlw"
Sep 09 06:51:39.510018 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:39.509969    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f9zfx" podStartSLOduration=4.379324012 podStartE2EDuration="22.509916878s" totalImagesPullingTime="18.130592866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:39.509372136 +0000 UTC m=+22.729613250" watchObservedRunningTime="2026-09-09 06:51:39.509916878 +0000 UTC m=+22.730157995"
Sep 09 06:51:40.179863 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:40.179744    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T06:51:39.46827436Z","UUID":"b7f72a97-ce5e-40c9-bfb3-73bd8b7d6c8c","Handler":null,"Name":"","Endpoint":""}
Sep 09 06:51:40.181545 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:40.181523    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 06:51:40.181679 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:40.181557    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 06:51:40.496220 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:40.496136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-428wq" event={"ID":"ee1df979-5286-453c-bb5c-68f35263ef3d","Type":"ContainerStarted","Data":"fcfef47a4970eefbebd6e621fa0bfdaf159431a65d48bd90214f6a92f6568b27"}
Sep 09 06:51:40.510336 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:40.510293    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-428wq" podStartSLOduration=5.798166149 podStartE2EDuration="23.51027503s" totalImagesPullingTime="17.712108881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:40.50997471 +0000 UTC m=+23.730215820" watchObservedRunningTime="2026-09-09 06:51:40.51027503 +0000 UTC m=+23.730516142"
Sep 09 06:51:41.227711 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:41.227627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:41.227963 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:41.227741    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:41.227963 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:41.227751    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:41.227963 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:41.227750    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:41.227963 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:41.227852    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:41.227963 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:41.227918    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:41.501789 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:41.501722    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 06:51:41.502170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:41.502129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"e26c44394b0748c1f64b86f319bf1635e77093eca6eaa7c290b4d8874ae98de3"}
Sep 09 06:51:41.504074 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:41.504049    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7" event={"ID":"8fcfdca1-f156-4eb3-be22-9a8d6be237a8","Type":"ContainerStarted","Data":"029599074e014f314d82d8dd631e14cc930a1ce7c30b5660cf19fab6b33c909e"}
Sep 09 06:51:43.224293 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.224262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:43.224725 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.224262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:43.224725 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:43.224385    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:43.224725 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:43.224463    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:43.224725 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.224268    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:43.224725 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:43.224546    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:43.512871 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.512294    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 06:51:43.513220 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.512915    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"af3c67b466251549c29c03146e4f2e9fc9f47f31cd9ecfa781ac52d32abd0f98"}
Sep 09 06:51:43.513616 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.513239    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:43.513616 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.513373    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:43.513616 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.513475    2562 scope.go:117] "RemoveContainer" containerID="c520abf08089debcc9f2aa88cb9f791e4632e172634360f98354b8a2227e4ee6"
Sep 09 06:51:43.513616 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.513495    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:43.533718 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.533641    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:43.535846 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.535627    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:51:43.548261 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:43.547720    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vwfz7" podStartSLOduration=6.584529612 podStartE2EDuration="26.547700087s" totalImagesPullingTime="19.963170475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:41.522656087 +0000 UTC m=+24.742897210" watchObservedRunningTime="2026-09-09 06:51:43.547700087 +0000 UTC m=+26.767941203"
Sep 09 06:51:44.518438 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:44.518412    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 06:51:44.519182 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:44.518784    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" event={"ID":"c8fe6d86-9c3c-45bb-ba16-713c85718256","Type":"ContainerStarted","Data":"f9f96c3191cc30bfbec7f83a47607905463b18b3433397f600ecbf327c640bcd"}
Sep 09 06:51:44.520652 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:44.520630    2562 generic.go:356] "Generic (PLEG): container finished" podID="24c3a8e5-a25d-4de8-a74d-bb19e329f6f1" containerID="392bf95eb4f9b23baf84e06356e9db56b6a22100a1e6ea72ce47878a6f2b1cf5" exitCode=0
Sep 09 06:51:44.520652 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:44.520659    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72zrg" event={"ID":"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1","Type":"ContainerDied","Data":"392bf95eb4f9b23baf84e06356e9db56b6a22100a1e6ea72ce47878a6f2b1cf5"}
Sep 09 06:51:44.547400 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:44.547347    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4" podStartSLOduration=9.811243894 podStartE2EDuration="27.547333152s" totalImagesPullingTime="17.736089258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:44.546251757 +0000 UTC m=+27.766492870" watchObservedRunningTime="2026-09-09 06:51:44.547333152 +0000 UTC m=+27.767574274"
Sep 09 06:51:45.223950 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.223786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:45.224169 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.223786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:45.224169 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:45.224020    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:45.224169 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.223802    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:45.224169 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:45.224096    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:45.224323 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:45.224200    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:45.312853 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.312826    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fct4f"]
Sep 09 06:51:45.316635 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.316531    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mfdk5"]
Sep 09 06:51:45.317173 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.317154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjl5t"]
Sep 09 06:51:45.522440 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.522408    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:45.522440 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.522429    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:45.522827 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:45.522410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:45.522827 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:45.522504    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:45.522827 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:45.522601    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:45.522934 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:45.522916    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:46.526112 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:46.526068    2562 generic.go:356] "Generic (PLEG): container finished" podID="24c3a8e5-a25d-4de8-a74d-bb19e329f6f1" containerID="5b155e431049eea833d058f4e650335d0c17c5b95098dbc935fab22bf08f9505" exitCode=0
Sep 09 06:51:46.526491 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:46.526137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72zrg" event={"ID":"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1","Type":"ContainerDied","Data":"5b155e431049eea833d058f4e650335d0c17c5b95098dbc935fab22bf08f9505"}
Sep 09 06:51:47.224921 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:47.224890    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:47.225068 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:47.224987    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:47.225068 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:47.225053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:47.225166 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:47.225149    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:47.225204 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:47.225192    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:47.225276 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:47.225260    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:48.533419 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:48.533248    2562 generic.go:356] "Generic (PLEG): container finished" podID="24c3a8e5-a25d-4de8-a74d-bb19e329f6f1" containerID="50c6411a2384c173e5af605a11220a331912db5edf66d7d5e2d9f32ebbc7b3ae" exitCode=0
Sep 09 06:51:48.533419 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:48.533330    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72zrg" event={"ID":"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1","Type":"ContainerDied","Data":"50c6411a2384c173e5af605a11220a331912db5edf66d7d5e2d9f32ebbc7b3ae"}
Sep 09 06:51:49.224745 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:49.224688    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:49.224924 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:49.224688    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:49.224924 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:49.224688    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:49.224924 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:49.224904    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fct4f" podUID="2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d"
Sep 09 06:51:49.225071 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:49.224938    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjl5t" podUID="9552e261-a17b-4376-99b3-5ff9f8f87b0e"
Sep 09 06:51:49.225071 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:49.224985    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfdk5" podUID="20f229be-0318-40fb-aa2a-62e0df181580"
Sep 09 06:51:50.963234 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:50.963197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:50.963712 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:50.963350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:50.963712 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:50.963371    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:50.963712 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:50.963457    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret podName:2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d nodeName:}" failed. No retries permitted until 2026-09-09 06:52:22.96343573 +0000 UTC m=+66.183676837 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret") pod "global-pull-secret-syncer-fct4f" (UID: "2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 06:51:50.963712 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:50.963479    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:50.963712 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:50.963534    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs podName:20f229be-0318-40fb-aa2a-62e0df181580 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:22.963516607 +0000 UTC m=+66.183757702 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs") pod "network-metrics-daemon-mfdk5" (UID: "20f229be-0318-40fb-aa2a-62e0df181580") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 06:51:51.063925 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.063890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:51.064136 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:51.064042    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 06:51:51.064136 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:51.064065    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 06:51:51.064136 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:51.064076    2562 projected.go:196] Error preparing data for projected volume kube-api-access-l2k9s for pod openshift-network-diagnostics/network-check-target-kjl5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:51.064293 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:51.064145    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s podName:9552e261-a17b-4376-99b3-5ff9f8f87b0e nodeName:}" failed. No retries permitted until 2026-09-09 06:52:23.064126722 +0000 UTC m=+66.284367815 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l2k9s" (UniqueName: "kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s") pod "network-check-target-kjl5t" (UID: "9552e261-a17b-4376-99b3-5ff9f8f87b0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 06:51:51.084354 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.084323    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-122.ec2.internal" event="NodeReady"
Sep 09 06:51:51.084501 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.084475    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 06:51:51.135182 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.135091    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7rj9m"]
Sep 09 06:51:51.159709 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.159682    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fkqdm"]
Sep 09 06:51:51.159867 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.159811    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.163672 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.163319    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 06:51:51.163672 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.163398    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n7krz\""
Sep 09 06:51:51.163672 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.163586    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 06:51:51.175441 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.175421    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rj9m"]
Sep 09 06:51:51.175521 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.175450    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fkqdm"]
Sep 09 06:51:51.175559 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.175519    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fkqdm"
Sep 09 06:51:51.178506 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.178484    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 06:51:51.178610 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.178536    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 06:51:51.178610 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.178536    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dk92v\""
Sep 09 06:51:51.179029 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.178971    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 06:51:51.224641 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.224612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:51:51.224806 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.224618    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:51:51.224806 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.224618    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:51:51.228215 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.228185    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxxvd\""
Sep 09 06:51:51.228344 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.228224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 06:51:51.228510 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.228454    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 06:51:51.228510 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.228475    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bgwvj\""
Sep 09 06:51:51.228640 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.228629    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 06:51:51.228716 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.228688    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 06:51:51.265335 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.265306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e258fb-04a0-45c1-82ad-7bde1452a792-cert\") pod \"ingress-canary-fkqdm\" (UID: \"57e258fb-04a0-45c1-82ad-7bde1452a792\") " pod="openshift-ingress-canary/ingress-canary-fkqdm"
Sep 09 06:51:51.265472 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.265348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-config-volume\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.265472 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.265373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.265472 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.265402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-tmp-dir\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.265472 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.265425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj52v\" (UniqueName: \"kubernetes.io/projected/57e258fb-04a0-45c1-82ad-7bde1452a792-kube-api-access-zj52v\") pod \"ingress-canary-fkqdm\" (UID: \"57e258fb-04a0-45c1-82ad-7bde1452a792\") " pod="openshift-ingress-canary/ingress-canary-fkqdm"
Sep 09 06:51:51.265643 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.265492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4tb\" (UniqueName: \"kubernetes.io/projected/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-kube-api-access-9z4tb\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.366584 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.366545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4tb\" (UniqueName: \"kubernetes.io/projected/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-kube-api-access-9z4tb\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.366769 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.366730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e258fb-04a0-45c1-82ad-7bde1452a792-cert\") pod \"ingress-canary-fkqdm\" (UID: \"57e258fb-04a0-45c1-82ad-7bde1452a792\") " pod="openshift-ingress-canary/ingress-canary-fkqdm"
Sep 09 06:51:51.366839 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.366761    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-config-volume\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.366839 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.366807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.366839 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.366826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-tmp-dir\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.366976 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.366848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj52v\" (UniqueName: \"kubernetes.io/projected/57e258fb-04a0-45c1-82ad-7bde1452a792-kube-api-access-zj52v\") pod \"ingress-canary-fkqdm\" (UID: \"57e258fb-04a0-45c1-82ad-7bde1452a792\") " pod="openshift-ingress-canary/ingress-canary-fkqdm"
Sep 09 06:51:51.367175 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:51.367145    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 06:51:51.367369 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:51.367235    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls podName:8f4ffda3-2dfb-47c8-b650-e7a7f7adc082 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:51.867213735 +0000 UTC m=+35.087454829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls") pod "dns-default-7rj9m" (UID: "8f4ffda3-2dfb-47c8-b650-e7a7f7adc082") : secret "dns-default-metrics-tls" not found
Sep 09 06:51:51.367369 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.367357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-tmp-dir\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.367575 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.367551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-config-volume\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.371427 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.371401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57e258fb-04a0-45c1-82ad-7bde1452a792-cert\") pod \"ingress-canary-fkqdm\" (UID: \"57e258fb-04a0-45c1-82ad-7bde1452a792\") " pod="openshift-ingress-canary/ingress-canary-fkqdm"
Sep 09 06:51:51.376821 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.376795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4tb\" (UniqueName: \"kubernetes.io/projected/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-kube-api-access-9z4tb\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.376949 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.376831    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj52v\" (UniqueName: \"kubernetes.io/projected/57e258fb-04a0-45c1-82ad-7bde1452a792-kube-api-access-zj52v\") pod \"ingress-canary-fkqdm\" (UID: \"57e258fb-04a0-45c1-82ad-7bde1452a792\") " pod="openshift-ingress-canary/ingress-canary-fkqdm"
Sep 09 06:51:51.483155 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.483091    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fkqdm"
Sep 09 06:51:51.871431 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:51.871339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:51.871617 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:51.871477    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 06:51:51.871617 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:51.871548    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls podName:8f4ffda3-2dfb-47c8-b650-e7a7f7adc082 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:52.871525004 +0000 UTC m=+36.091766109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls") pod "dns-default-7rj9m" (UID: "8f4ffda3-2dfb-47c8-b650-e7a7f7adc082") : secret "dns-default-metrics-tls" not found
Sep 09 06:51:52.879032 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:52.878996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:52.879692 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:52.879145    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 06:51:52.879692 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:52.879195    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls podName:8f4ffda3-2dfb-47c8-b650-e7a7f7adc082 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:54.879181551 +0000 UTC m=+38.099422648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls") pod "dns-default-7rj9m" (UID: "8f4ffda3-2dfb-47c8-b650-e7a7f7adc082") : secret "dns-default-metrics-tls" not found
Sep 09 06:51:53.028574 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.028499    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fkqdm"]
Sep 09 06:51:53.032319 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:53.032295    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e258fb_04a0_45c1_82ad_7bde1452a792.slice/crio-01042569dcfd34f8f722aaaa00a0e103396635f596c0f9535227c0f8137c3a31 WatchSource:0}: Error finding container 01042569dcfd34f8f722aaaa00a0e103396635f596c0f9535227c0f8137c3a31: Status 404 returned error can't find the container with id 01042569dcfd34f8f722aaaa00a0e103396635f596c0f9535227c0f8137c3a31
Sep 09 06:51:53.367655 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.367620    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh"]
Sep 09 06:51:53.384018 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.383988    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh"]
Sep 09 06:51:53.384186 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.384132    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh"
Sep 09 06:51:53.387118 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.387074    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 06:51:53.387228 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.387130    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 06:51:53.387655 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.387637    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9r7jf\""
Sep 09 06:51:53.464443 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.464384    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"]
Sep 09 06:51:53.484205 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.484167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmx9x\" (UniqueName: \"kubernetes.io/projected/31c01229-001d-46e7-ae61-1d931257e7b6-kube-api-access-tmx9x\") pod \"volume-data-source-validator-695c895bd6-p6bkh\" (UID: \"31c01229-001d-46e7-ae61-1d931257e7b6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh"
Sep 09 06:51:53.489865 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.489836    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qc9k2"]
Sep 09 06:51:53.490012 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.489959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.492897 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.492872    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 06:51:53.493031 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.492948    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8ckdw\""
Sep 09 06:51:53.493031 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.492974    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 06:51:53.493179 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.493158    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 06:51:53.493642 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.493621    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 06:51:53.521968 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.521942    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64b5bc69fb-rlpgt"]
Sep 09 06:51:53.522124 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.522087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.525139 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.525095    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 06:51:53.525283 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.525200    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h77qc\""
Sep 09 06:51:53.525405 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.525377    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 06:51:53.525488 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.525416    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 06:51:53.525488 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.525420    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 06:51:53.531878 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.531856    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 06:51:53.538352 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.538264    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"]
Sep 09 06:51:53.538450 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.538372    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qc9k2"]
Sep 09 06:51:53.538450 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.538375    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.538450 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.538394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64b5bc69fb-rlpgt"]
Sep 09 06:51:53.541181 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.541161    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 06:51:53.541286 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.541166    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 06:51:53.541286 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.541238    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 06:51:53.541661 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.541642    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4qwcc\""
Sep 09 06:51:53.546648 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.546618    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 06:51:53.549855 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.549832    2562 generic.go:356] "Generic (PLEG): container finished" podID="24c3a8e5-a25d-4de8-a74d-bb19e329f6f1" containerID="44124def4b882ccdf762a66336314a46083405af92d5a94e69c8a9a1c6f5cb45" exitCode=0
Sep 09 06:51:53.549955 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.549910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72zrg" event={"ID":"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1","Type":"ContainerDied","Data":"44124def4b882ccdf762a66336314a46083405af92d5a94e69c8a9a1c6f5cb45"}
Sep 09 06:51:53.551605 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.551574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fkqdm" event={"ID":"57e258fb-04a0-45c1-82ad-7bde1452a792","Type":"ContainerStarted","Data":"01042569dcfd34f8f722aaaa00a0e103396635f596c0f9535227c0f8137c3a31"}
Sep 09 06:51:53.585390 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.585358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmx9x\" (UniqueName: \"kubernetes.io/projected/31c01229-001d-46e7-ae61-1d931257e7b6-kube-api-access-tmx9x\") pod \"volume-data-source-validator-695c895bd6-p6bkh\" (UID: \"31c01229-001d-46e7-ae61-1d931257e7b6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh"
Sep 09 06:51:53.585485 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.585411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7796c\" (UniqueName: \"kubernetes.io/projected/5e13bfba-38d5-4775-9a82-b74c70758943-kube-api-access-7796c\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.585485 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.585470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e13bfba-38d5-4775-9a82-b74c70758943-config\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.585565 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.585542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e13bfba-38d5-4775-9a82-b74c70758943-serving-cert\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.595512 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.595490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmx9x\" (UniqueName: \"kubernetes.io/projected/31c01229-001d-46e7-ae61-1d931257e7b6-kube-api-access-tmx9x\") pod \"volume-data-source-validator-695c895bd6-p6bkh\" (UID: \"31c01229-001d-46e7-ae61-1d931257e7b6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh"
Sep 09 06:51:53.686361 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-tmp\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.686539 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.686539 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa15c0ca-032c-4568-ac2f-94a73a870ddf-ca-trust-extracted\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.686539 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e13bfba-38d5-4775-9a82-b74c70758943-serving-cert\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.686539 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-image-registry-private-configuration\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.686539 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-trusted-ca\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.686782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-installation-pull-secrets\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.686782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-bound-sa-token\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.686782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-serving-cert\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.686782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrtv\" (UniqueName: \"kubernetes.io/projected/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-kube-api-access-rrrtv\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.686782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7796c\" (UniqueName: \"kubernetes.io/projected/5e13bfba-38d5-4775-9a82-b74c70758943-kube-api-access-7796c\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.686782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.686782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.687248 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e13bfba-38d5-4775-9a82-b74c70758943-config\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.687248 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-snapshots\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.687248 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-certificates\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.687248 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.686970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lhl\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-kube-api-access-f7lhl\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.687591 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.687568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e13bfba-38d5-4775-9a82-b74c70758943-config\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.689290 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.689265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e13bfba-38d5-4775-9a82-b74c70758943-serving-cert\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.693062 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.693041    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh"
Sep 09 06:51:53.711451 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.711386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7796c\" (UniqueName: \"kubernetes.io/projected/5e13bfba-38d5-4775-9a82-b74c70758943-kube-api-access-7796c\") pod \"service-ca-operator-859f94c4b-52mjj\" (UID: \"5e13bfba-38d5-4775-9a82-b74c70758943\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.787462 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrtv\" (UniqueName: \"kubernetes.io/projected/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-kube-api-access-rrrtv\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.787761 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.787761 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.787761 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-snapshots\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.788001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-certificates\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.788001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lhl\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-kube-api-access-f7lhl\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.788001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-tmp\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.788001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.788001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa15c0ca-032c-4568-ac2f-94a73a870ddf-ca-trust-extracted\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.788001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-image-registry-private-configuration\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.788001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.787991    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-trusted-ca\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.788552 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.788019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-installation-pull-secrets\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.788552 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.788043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-bound-sa-token\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.788552 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.788089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-serving-cert\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.788761 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:53.788596    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 06:51:53.788761 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:53.788617    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5bc69fb-rlpgt: secret "image-registry-tls" not found
Sep 09 06:51:53.788761 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:53.788680    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls podName:fa15c0ca-032c-4568-ac2f-94a73a870ddf nodeName:}" failed. No retries permitted until 2026-09-09 06:51:54.288659747 +0000 UTC m=+37.508900858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls") pod "image-registry-64b5bc69fb-rlpgt" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf") : secret "image-registry-tls" not found
Sep 09 06:51:53.788957 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.788777    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.790001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.789167    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-certificates\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.790001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.789258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.790001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.789424    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-tmp\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.790001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.789553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa15c0ca-032c-4568-ac2f-94a73a870ddf-ca-trust-extracted\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.790001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.789577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-trusted-ca\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.790001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.789958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-snapshots\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.791160 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.791116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-serving-cert\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.791865 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.791842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-installation-pull-secrets\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.792653 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.792634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-image-registry-private-configuration\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.797967 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.797944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lhl\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-kube-api-access-f7lhl\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.798091 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.798074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrtv\" (UniqueName: \"kubernetes.io/projected/bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6-kube-api-access-rrrtv\") pod \"insights-operator-76bcd6ccf5-qc9k2\" (UID: \"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.798651 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.798606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"
Sep 09 06:51:53.798946 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.798916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-bound-sa-token\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:53.833772 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.833673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2"
Sep 09 06:51:53.977398 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:53.977241    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh"]
Sep 09 06:51:53.979801 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:53.979770    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c01229_001d_46e7_ae61_1d931257e7b6.slice/crio-59b8861806666491f7f5c220ade96443b33cd8c48f4ba240a45525aab5dac60a WatchSource:0}: Error finding container 59b8861806666491f7f5c220ade96443b33cd8c48f4ba240a45525aab5dac60a: Status 404 returned error can't find the container with id 59b8861806666491f7f5c220ade96443b33cd8c48f4ba240a45525aab5dac60a
Sep 09 06:51:54.082692 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.082615    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj"]
Sep 09 06:51:54.085586 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:54.085562    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e13bfba_38d5_4775_9a82_b74c70758943.slice/crio-15e6119b952f496deb721abe88dbeb95268f0d016ee514903d9244755f817429 WatchSource:0}: Error finding container 15e6119b952f496deb721abe88dbeb95268f0d016ee514903d9244755f817429: Status 404 returned error can't find the container with id 15e6119b952f496deb721abe88dbeb95268f0d016ee514903d9244755f817429
Sep 09 06:51:54.126174 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.126142    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qc9k2"]
Sep 09 06:51:54.129639 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:51:54.129607    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb894c77_b7fc_4c06_b4fe_b5c9c9bea8e6.slice/crio-fd85cdd3ec02431ff6452b3cdf82c890128841d106bdc9240a4abc0cdbb5565d WatchSource:0}: Error finding container fd85cdd3ec02431ff6452b3cdf82c890128841d106bdc9240a4abc0cdbb5565d: Status 404 returned error can't find the container with id fd85cdd3ec02431ff6452b3cdf82c890128841d106bdc9240a4abc0cdbb5565d
Sep 09 06:51:54.292776 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.292737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:54.292935 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:54.292909    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 06:51:54.292991 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:54.292934    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5bc69fb-rlpgt: secret "image-registry-tls" not found
Sep 09 06:51:54.293053 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:54.293001    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls podName:fa15c0ca-032c-4568-ac2f-94a73a870ddf nodeName:}" failed. No retries permitted until 2026-09-09 06:51:55.292985352 +0000 UTC m=+38.513226442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls") pod "image-registry-64b5bc69fb-rlpgt" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf") : secret "image-registry-tls" not found
Sep 09 06:51:54.555748 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.555674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh" event={"ID":"31c01229-001d-46e7-ae61-1d931257e7b6","Type":"ContainerStarted","Data":"59b8861806666491f7f5c220ade96443b33cd8c48f4ba240a45525aab5dac60a"}
Sep 09 06:51:54.557057 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.557022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2" event={"ID":"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6","Type":"ContainerStarted","Data":"fd85cdd3ec02431ff6452b3cdf82c890128841d106bdc9240a4abc0cdbb5565d"}
Sep 09 06:51:54.561389 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.561349    2562 generic.go:356] "Generic (PLEG): container finished" podID="24c3a8e5-a25d-4de8-a74d-bb19e329f6f1" containerID="ee3cde3f748e80f19300db9b3ec7cfbb13b7bff376a00ca00bc4c7c46954c9f6" exitCode=0
Sep 09 06:51:54.561503 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.561433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72zrg" event={"ID":"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1","Type":"ContainerDied","Data":"ee3cde3f748e80f19300db9b3ec7cfbb13b7bff376a00ca00bc4c7c46954c9f6"}
Sep 09 06:51:54.562772 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.562743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj" event={"ID":"5e13bfba-38d5-4775-9a82-b74c70758943","Type":"ContainerStarted","Data":"15e6119b952f496deb721abe88dbeb95268f0d016ee514903d9244755f817429"}
Sep 09 06:51:54.897064 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:54.896994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:54.897241 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:54.897186    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 06:51:54.897313 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:54.897265    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls podName:8f4ffda3-2dfb-47c8-b650-e7a7f7adc082 nodeName:}" failed. No retries permitted until 2026-09-09 06:51:58.897244831 +0000 UTC m=+42.117485938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls") pod "dns-default-7rj9m" (UID: "8f4ffda3-2dfb-47c8-b650-e7a7f7adc082") : secret "dns-default-metrics-tls" not found
Sep 09 06:51:55.300052 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:55.299954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:55.300527 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:55.300089    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 06:51:55.300527 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:55.300130    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5bc69fb-rlpgt: secret "image-registry-tls" not found
Sep 09 06:51:55.300527 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:55.300194    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls podName:fa15c0ca-032c-4568-ac2f-94a73a870ddf nodeName:}" failed. No retries permitted until 2026-09-09 06:51:57.300173919 +0000 UTC m=+40.520415022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls") pod "image-registry-64b5bc69fb-rlpgt" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf") : secret "image-registry-tls" not found
Sep 09 06:51:57.317491 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.317447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:51:57.317868 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:57.317635    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 06:51:57.317868 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:57.317658    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5bc69fb-rlpgt: secret "image-registry-tls" not found
Sep 09 06:51:57.317868 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:57.317730    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls podName:fa15c0ca-032c-4568-ac2f-94a73a870ddf nodeName:}" failed. No retries permitted until 2026-09-09 06:52:01.31770936 +0000 UTC m=+44.537950458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls") pod "image-registry-64b5bc69fb-rlpgt" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf") : secret "image-registry-tls" not found
Sep 09 06:51:57.573563 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.573520    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fkqdm" event={"ID":"57e258fb-04a0-45c1-82ad-7bde1452a792","Type":"ContainerStarted","Data":"8c32ecf29b8efbb0fafb41847e7e6528d2a3781252ad11f9b34f24915f63a923"}
Sep 09 06:51:57.575319 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.575285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh" event={"ID":"31c01229-001d-46e7-ae61-1d931257e7b6","Type":"ContainerStarted","Data":"784495eb5f828d0c2c03bcd5d0cd65c87b0dfd58d4105c99b7079475f6d4712e"}
Sep 09 06:51:57.576996 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.576968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2" event={"ID":"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6","Type":"ContainerStarted","Data":"e8b47a4edb069a66d21126442519be9367f0755dd3b5a4857ea6275a9f446af4"}
Sep 09 06:51:57.580870 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.580847    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72zrg" event={"ID":"24c3a8e5-a25d-4de8-a74d-bb19e329f6f1","Type":"ContainerStarted","Data":"3605314e3a962e8fce67e8c6230d2dc72d5c8f80bfc8e2aa3ce3ef1a01ea9a0a"}
Sep 09 06:51:57.589943 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.589895    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fkqdm" podStartSLOduration=2.867905527 podStartE2EDuration="6.589880108s" totalImagesPullingTime="3.721974581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:57.589689387 +0000 UTC m=+40.809930500" watchObservedRunningTime="2026-09-09 06:51:57.589880108 +0000 UTC m=+40.810121223"
Sep 09 06:51:57.605182 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.605130    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-p6bkh" podStartSLOduration=1.8294520109999999 podStartE2EDuration="4.605082369s" totalImagesPullingTime="2.775630358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:57.604629657 +0000 UTC m=+40.824870770" watchObservedRunningTime="2026-09-09 06:51:57.605082369 +0000 UTC m=+40.825323483"
Sep 09 06:51:57.627375 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.627325    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2" podStartSLOduration=1.672477057 podStartE2EDuration="4.62730589s" totalImagesPullingTime="2.954828833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:57.626226657 +0000 UTC m=+40.846467770" watchObservedRunningTime="2026-09-09 06:51:57.62730589 +0000 UTC m=+40.847547005"
Sep 09 06:51:57.654037 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:57.653981    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-72zrg" podStartSLOduration=10.867373378 podStartE2EDuration="40.653961026s" totalImagesPullingTime="29.786587648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:57.653120228 +0000 UTC m=+40.873361372" watchObservedRunningTime="2026-09-09 06:51:57.653961026 +0000 UTC m=+40.874202139"
Sep 09 06:51:58.585313 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:58.585275    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj" event={"ID":"5e13bfba-38d5-4775-9a82-b74c70758943","Type":"ContainerStarted","Data":"2310dc59e87491c114ad78fe8d32c9079d892b5597061dfa88d8efba80051507"}
Sep 09 06:51:58.601216 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:58.601176    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj" podStartSLOduration=1.261750385 podStartE2EDuration="5.601161536s" totalImagesPullingTime="4.339411151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:51:58.601078169 +0000 UTC m=+41.821319278" watchObservedRunningTime="2026-09-09 06:51:58.601161536 +0000 UTC m=+41.821402645"
Sep 09 06:51:58.933783 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:58.933740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:51:58.933958 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:58.933886    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 06:51:58.933958 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:51:58.933957    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls podName:8f4ffda3-2dfb-47c8-b650-e7a7f7adc082 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:06.933939225 +0000 UTC m=+50.154180321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls") pod "dns-default-7rj9m" (UID: "8f4ffda3-2dfb-47c8-b650-e7a7f7adc082") : secret "dns-default-metrics-tls" not found
Sep 09 06:51:59.873903 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:59.873853    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t555s"]
Sep 09 06:51:59.895680 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:59.895653    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t555s"]
Sep 09 06:51:59.895805 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:59.895766    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:51:59.898457 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:59.898431    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 06:51:59.898457 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:59.898437    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nqhdr\""
Sep 09 06:51:59.898748 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:51:59.898734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 06:52:00.042179 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.042145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.042344 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.042218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwq7\" (UniqueName: \"kubernetes.io/projected/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-api-access-rzwq7\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.042344 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.042287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-crio-socket\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.042344 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.042318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.042467 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.042367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-data-volume\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.143754 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.143675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-crio-socket\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.143754 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.143716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.143889 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.143773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-data-volume\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.143889 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.143815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.143958 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.143891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-crio-socket\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.143958 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:00.143909    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:00.143958 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.143926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzwq7\" (UniqueName: \"kubernetes.io/projected/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-api-access-rzwq7\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.143958 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:00.143956    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 06:52:00.144076 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:00.143979    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:00.643959909 +0000 UTC m=+43.864201008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:00.144076 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:00.144005    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:00.643992787 +0000 UTC m=+43.864233894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : configmap "kube-rbac-proxy" not found
Sep 09 06:52:00.144185 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.144162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-data-volume\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.152749 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.152729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzwq7\" (UniqueName: \"kubernetes.io/projected/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-api-access-rzwq7\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.647053 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.647018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.647240 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:00.647156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:00.647240 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:00.647227    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:00.647348 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:00.647268    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 06:52:00.647348 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:00.647278    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:01.647265636 +0000 UTC m=+44.867506726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:00.647348 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:00.647323    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:01.64731151 +0000 UTC m=+44.867552602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : configmap "kube-rbac-proxy" not found
Sep 09 06:52:01.353215 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.353175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:52:01.353603 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:01.353292    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 06:52:01.353603 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:01.353304    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5bc69fb-rlpgt: secret "image-registry-tls" not found
Sep 09 06:52:01.353603 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:01.353350    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls podName:fa15c0ca-032c-4568-ac2f-94a73a870ddf nodeName:}" failed. No retries permitted until 2026-09-09 06:52:09.353338078 +0000 UTC m=+52.573579183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls") pod "image-registry-64b5bc69fb-rlpgt" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf") : secret "image-registry-tls" not found
Sep 09 06:52:01.619178 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.619090    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cp5xk_3cd0276a-00b2-49c9-86a9-4bd63ccf9457/dns-node-resolver/0.log"
Sep 09 06:52:01.655509 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.655476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:01.655662 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.655541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:01.655662 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:01.655645    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:01.655662 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:01.655656    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 06:52:01.655755 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:01.655713    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:03.655693383 +0000 UTC m=+46.875934477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:01.655755 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:01.655729    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:03.655723285 +0000 UTC m=+46.875964377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : configmap "kube-rbac-proxy" not found
Sep 09 06:52:01.947451 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.947410    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n7pts"]
Sep 09 06:52:01.966732 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.966705    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n7pts"]
Sep 09 06:52:01.966896 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.966813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:01.969725 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.969700    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 06:52:01.969725 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.969706    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 06:52:01.969909 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.969754    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 06:52:01.970542 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.970517    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4s6zn\""
Sep 09 06:52:01.970542 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:01.970530    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 06:52:02.160346 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.160307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d27c01eb-9cde-4d91-ad2b-83cba31b104b-signing-cabundle\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.160517 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.160444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7bdl\" (UniqueName: \"kubernetes.io/projected/d27c01eb-9cde-4d91-ad2b-83cba31b104b-kube-api-access-p7bdl\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.160517 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.160475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d27c01eb-9cde-4d91-ad2b-83cba31b104b-signing-key\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.260950 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.260865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7bdl\" (UniqueName: \"kubernetes.io/projected/d27c01eb-9cde-4d91-ad2b-83cba31b104b-kube-api-access-p7bdl\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.260950 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.260903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d27c01eb-9cde-4d91-ad2b-83cba31b104b-signing-key\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.261174 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.261053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d27c01eb-9cde-4d91-ad2b-83cba31b104b-signing-cabundle\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.261673 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.261655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d27c01eb-9cde-4d91-ad2b-83cba31b104b-signing-cabundle\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.263372 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.263356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d27c01eb-9cde-4d91-ad2b-83cba31b104b-signing-key\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.269512 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.269491    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7bdl\" (UniqueName: \"kubernetes.io/projected/d27c01eb-9cde-4d91-ad2b-83cba31b104b-kube-api-access-p7bdl\") pod \"service-ca-75d99c484-n7pts\" (UID: \"d27c01eb-9cde-4d91-ad2b-83cba31b104b\") " pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.275429 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.275411    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-n7pts"
Sep 09 06:52:02.561073 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.561000    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-n7pts"]
Sep 09 06:52:02.563977 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:52:02.563951    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27c01eb_9cde_4d91_ad2b_83cba31b104b.slice/crio-21fd866c04bf1b9726250a0a4543086193fb1b6966f642c3d1ab0412e1353f42 WatchSource:0}: Error finding container 21fd866c04bf1b9726250a0a4543086193fb1b6966f642c3d1ab0412e1353f42: Status 404 returned error can't find the container with id 21fd866c04bf1b9726250a0a4543086193fb1b6966f642c3d1ab0412e1353f42
Sep 09 06:52:02.595648 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.595616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-n7pts" event={"ID":"d27c01eb-9cde-4d91-ad2b-83cba31b104b","Type":"ContainerStarted","Data":"21fd866c04bf1b9726250a0a4543086193fb1b6966f642c3d1ab0412e1353f42"}
Sep 09 06:52:02.824414 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:02.824340    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fjhls_9e27fe24-fa2b-4e20-869b-8c520d98c116/node-ca/0.log"
Sep 09 06:52:03.420496 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:03.420460    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fkqdm_57e258fb-04a0-45c1-82ad-7bde1452a792/serve-healthcheck-canary/0.log"
Sep 09 06:52:03.601995 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:03.601953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-n7pts" event={"ID":"d27c01eb-9cde-4d91-ad2b-83cba31b104b","Type":"ContainerStarted","Data":"0aa29867f983603f9996b31fa62b9a9db6d1ad19e7ab05890188e323d491f84a"}
Sep 09 06:52:03.621541 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:03.621486    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-n7pts" podStartSLOduration=2.6214632570000003 podStartE2EDuration="2.621463257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:03.619534836 +0000 UTC m=+46.839775966" watchObservedRunningTime="2026-09-09 06:52:03.621463257 +0000 UTC m=+46.841704372"
Sep 09 06:52:03.673888 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:03.673798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:03.674060 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:03.673908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:03.674146 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:03.674062    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:03.674146 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:03.674143    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:07.674124681 +0000 UTC m=+50.894365789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:03.674743 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:03.674721    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 06:52:03.674814 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:03.674779    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:07.674763469 +0000 UTC m=+50.895004574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : configmap "kube-rbac-proxy" not found
Sep 09 06:52:06.998142 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:06.998092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:52:06.998496 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:06.998248    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 06:52:06.998496 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:06.998309    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls podName:8f4ffda3-2dfb-47c8-b650-e7a7f7adc082 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:22.99829276 +0000 UTC m=+66.218533856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls") pod "dns-default-7rj9m" (UID: "8f4ffda3-2dfb-47c8-b650-e7a7f7adc082") : secret "dns-default-metrics-tls" not found
Sep 09 06:52:07.703023 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:07.702983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:07.703219 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:07.703096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:07.703219 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:07.703138    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:07.703219 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:07.703184    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 06:52:07.703219 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:07.703214    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:15.703198626 +0000 UTC m=+58.923439717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : secret "insights-runtime-extractor-tls" not found
Sep 09 06:52:07.703373 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:07.703247    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm podName:5204bfd0-c411-4659-b456-7f7e9ea0c5c6 nodeName:}" failed. No retries permitted until 2026-09-09 06:52:15.703227769 +0000 UTC m=+58.923468868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t555s" (UID: "5204bfd0-c411-4659-b456-7f7e9ea0c5c6") : configmap "kube-rbac-proxy" not found
Sep 09 06:52:09.415838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:09.415806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:52:09.418201 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:09.418181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"image-registry-64b5bc69fb-rlpgt\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") " pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:52:09.449094 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:09.449067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:52:09.718067 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:09.717998    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64b5bc69fb-rlpgt"]
Sep 09 06:52:09.721161 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:52:09.721140    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa15c0ca_032c_4568_ac2f_94a73a870ddf.slice/crio-f33046970ea6d436c08d63c18f37e1367121d5e75f06e6e519560ba3df6c7c4c WatchSource:0}: Error finding container f33046970ea6d436c08d63c18f37e1367121d5e75f06e6e519560ba3df6c7c4c: Status 404 returned error can't find the container with id f33046970ea6d436c08d63c18f37e1367121d5e75f06e6e519560ba3df6c7c4c
Sep 09 06:52:10.628976 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:10.628936    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt" event={"ID":"fa15c0ca-032c-4568-ac2f-94a73a870ddf","Type":"ContainerStarted","Data":"cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890"}
Sep 09 06:52:10.629418 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:10.629032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt" event={"ID":"fa15c0ca-032c-4568-ac2f-94a73a870ddf","Type":"ContainerStarted","Data":"f33046970ea6d436c08d63c18f37e1367121d5e75f06e6e519560ba3df6c7c4c"}
Sep 09 06:52:10.629418 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:10.629121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:52:10.649595 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:10.649549    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt" podStartSLOduration=17.649536082 podStartE2EDuration="17.649536082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:10.648977646 +0000 UTC m=+53.869218759" watchObservedRunningTime="2026-09-09 06:52:10.649536082 +0000 UTC m=+53.869777194"
Sep 09 06:52:15.532939 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:15.532912    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnrr4"
Sep 09 06:52:15.767847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:15.767807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:15.768021 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:15.767951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:15.768439 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:15.768421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:15.770049 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:15.770023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5204bfd0-c411-4659-b456-7f7e9ea0c5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t555s\" (UID: \"5204bfd0-c411-4659-b456-7f7e9ea0c5c6\") " pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:15.805236 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:15.805165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t555s"
Sep 09 06:52:16.072955 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:16.072884    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t555s"]
Sep 09 06:52:16.075735 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:52:16.075712    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5204bfd0_c411_4659_b456_7f7e9ea0c5c6.slice/crio-9984f841d1fd42258ee6a55d42588798fd5e66ea1cb4a81bab8b038d597a0a06 WatchSource:0}: Error finding container 9984f841d1fd42258ee6a55d42588798fd5e66ea1cb4a81bab8b038d597a0a06: Status 404 returned error can't find the container with id 9984f841d1fd42258ee6a55d42588798fd5e66ea1cb4a81bab8b038d597a0a06
Sep 09 06:52:16.653011 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:16.652978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t555s" event={"ID":"5204bfd0-c411-4659-b456-7f7e9ea0c5c6","Type":"ContainerStarted","Data":"fa8e9a880fbd4857f09b4c248e0c296450cf8b4d9b6d17ba0bbe1f1470b8028c"}
Sep 09 06:52:16.653429 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:16.653035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t555s" event={"ID":"5204bfd0-c411-4659-b456-7f7e9ea0c5c6","Type":"ContainerStarted","Data":"9984f841d1fd42258ee6a55d42588798fd5e66ea1cb4a81bab8b038d597a0a06"}
Sep 09 06:52:17.658998 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:17.658960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t555s" event={"ID":"5204bfd0-c411-4659-b456-7f7e9ea0c5c6","Type":"ContainerStarted","Data":"e1b12d5010b077e33e75c53e2919e7ea606738f4142234f4aacd93120321953a"}
Sep 09 06:52:18.664349 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:18.664261    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t555s" event={"ID":"5204bfd0-c411-4659-b456-7f7e9ea0c5c6","Type":"ContainerStarted","Data":"41cce3b1e6eb2c6bb6a1561db92f864b746aa0f388668684ed9bc3f75b0c5849"}
Sep 09 06:52:18.683523 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:18.683470    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t555s" podStartSLOduration=17.633668319999998 podStartE2EDuration="19.683453814s" totalImagesPullingTime="2.049785494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:18.682305455 +0000 UTC m=+61.902546569" watchObservedRunningTime="2026-09-09 06:52:18.683453814 +0000 UTC m=+61.903694924"
Sep 09 06:52:23.018428 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.018388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:52:23.018857 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.018450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:52:23.018857 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.018485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:52:23.021360 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.021341    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 06:52:23.022171 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.022153    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 06:52:23.023647 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.023629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f4ffda3-2dfb-47c8-b650-e7a7f7adc082-metrics-tls\") pod \"dns-default-7rj9m\" (UID: \"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082\") " pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:52:23.031148 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.031127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d-original-pull-secret\") pod \"global-pull-secret-syncer-fct4f\" (UID: \"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d\") " pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:52:23.031218 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.031187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f229be-0318-40fb-aa2a-62e0df181580-metrics-certs\") pod \"network-metrics-daemon-mfdk5\" (UID: \"20f229be-0318-40fb-aa2a-62e0df181580\") " pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:52:23.042360 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.042342    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fct4f"
Sep 09 06:52:23.050290 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.050272    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bgwvj\""
Sep 09 06:52:23.058620 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.058599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfdk5"
Sep 09 06:52:23.120044 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.119985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:52:23.128066 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.128041    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 06:52:23.133499 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.133463    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 06:52:23.144211 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.144191    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2k9s\" (UniqueName: \"kubernetes.io/projected/9552e261-a17b-4376-99b3-5ff9f8f87b0e-kube-api-access-l2k9s\") pod \"network-check-target-kjl5t\" (UID: \"9552e261-a17b-4376-99b3-5ff9f8f87b0e\") " pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:52:23.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.274574    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n7krz\""
Sep 09 06:52:23.282125 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.282090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:52:23.315834 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.315807    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fct4f"]
Sep 09 06:52:23.318034 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:52:23.318008    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3268e0_c1e5_45b0_8f4c_81ac05e39d8d.slice/crio-7651a968cb319faec030319b003824e1397b0ca89ce94a858dd20123f261dea7 WatchSource:0}: Error finding container 7651a968cb319faec030319b003824e1397b0ca89ce94a858dd20123f261dea7: Status 404 returned error can't find the container with id 7651a968cb319faec030319b003824e1397b0ca89ce94a858dd20123f261dea7
Sep 09 06:52:23.338957 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.338121    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mfdk5"]
Sep 09 06:52:23.344875 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.339456    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxxvd\""
Sep 09 06:52:23.346440 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.346261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:52:23.557039 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.556964    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rj9m"]
Sep 09 06:52:23.559492 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:52:23.559470    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4ffda3_2dfb_47c8_b650_e7a7f7adc082.slice/crio-d7c4e907c7344f41b26c765fdbdfb9ff3be4b5168a5373432a62dedb565bb1ac WatchSource:0}: Error finding container d7c4e907c7344f41b26c765fdbdfb9ff3be4b5168a5373432a62dedb565bb1ac: Status 404 returned error can't find the container with id d7c4e907c7344f41b26c765fdbdfb9ff3be4b5168a5373432a62dedb565bb1ac
Sep 09 06:52:23.629764 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.629734    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjl5t"]
Sep 09 06:52:23.633749 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:52:23.633728    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9552e261_a17b_4376_99b3_5ff9f8f87b0e.slice/crio-c77e36dfce47de43c9378c3be234a54ba4b7b4fd5b5ec206806cac4ed37f7578 WatchSource:0}: Error finding container c77e36dfce47de43c9378c3be234a54ba4b7b4fd5b5ec206806cac4ed37f7578: Status 404 returned error can't find the container with id c77e36dfce47de43c9378c3be234a54ba4b7b4fd5b5ec206806cac4ed37f7578
Sep 09 06:52:23.680767 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.680734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fct4f" event={"ID":"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d","Type":"ContainerStarted","Data":"7651a968cb319faec030319b003824e1397b0ca89ce94a858dd20123f261dea7"}
Sep 09 06:52:23.681929 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.681892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rj9m" event={"ID":"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082","Type":"ContainerStarted","Data":"d7c4e907c7344f41b26c765fdbdfb9ff3be4b5168a5373432a62dedb565bb1ac"}
Sep 09 06:52:23.683069 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.683038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjl5t" event={"ID":"9552e261-a17b-4376-99b3-5ff9f8f87b0e","Type":"ContainerStarted","Data":"c77e36dfce47de43c9378c3be234a54ba4b7b4fd5b5ec206806cac4ed37f7578"}
Sep 09 06:52:23.684149 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:23.684126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfdk5" event={"ID":"20f229be-0318-40fb-aa2a-62e0df181580","Type":"ContainerStarted","Data":"9236850cf231a427732c5bfde5ec2f5e942e70a8bbf0277ce48d25cffc670bf9"}
Sep 09 06:52:24.796806 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.795506    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"]
Sep 09 06:52:24.799622 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.798147    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.806798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.804456    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 06:52:24.806798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.804672    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 06:52:24.806798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.805445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 06:52:24.806798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.805636    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 06:52:24.806798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.805825    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 06:52:24.806798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.806011    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 06:52:24.806798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.806236    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 06:52:24.832683 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.832651    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"]
Sep 09 06:52:24.833300 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.832976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpd55\" (UniqueName: \"kubernetes.io/projected/2094e263-ae7e-437d-ad57-2bc058bb72bd-kube-api-access-lpd55\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.833300 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.833025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2094e263-ae7e-437d-ad57-2bc058bb72bd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.833300 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.833055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-ca\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.833300 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.833127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.833300 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.833157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.833300 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.833182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-hub\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.865864 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.864768    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64b5bc69fb-rlpgt"]
Sep 09 06:52:24.873612 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.873576    2562 patch_prober.go:28] interesting pod/image-registry-64b5bc69fb-rlpgt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 06:52:24.873759 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.873633    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt" podUID="fa15c0ca-032c-4568-ac2f-94a73a870ddf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 06:52:24.934492 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.934464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.934633 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.934511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.934633 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.934541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-hub\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.934633 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.934584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpd55\" (UniqueName: \"kubernetes.io/projected/2094e263-ae7e-437d-ad57-2bc058bb72bd-kube-api-access-lpd55\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.934633 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.934619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2094e263-ae7e-437d-ad57-2bc058bb72bd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.934898 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.934648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-ca\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.936733 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.936706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2094e263-ae7e-437d-ad57-2bc058bb72bd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.939716 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.939674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.939892 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.939867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-ca\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.939967 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.939946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-hub\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.940833 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.940812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2094e263-ae7e-437d-ad57-2bc058bb72bd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:24.962658 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:24.962606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpd55\" (UniqueName: \"kubernetes.io/projected/2094e263-ae7e-437d-ad57-2bc058bb72bd-kube-api-access-lpd55\") pod \"cluster-proxy-proxy-agent-65d75c7f68-lsb2w\" (UID: \"2094e263-ae7e-437d-ad57-2bc058bb72bd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:25.122304 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:25.122264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"
Sep 09 06:52:25.429382 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:25.429309    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w"]
Sep 09 06:52:25.695454 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:25.695360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfdk5" event={"ID":"20f229be-0318-40fb-aa2a-62e0df181580","Type":"ContainerStarted","Data":"988e47423df0bb3530efe7ac5fd87aa978a1dbca913c12d29078a94ad56d39f8"}
Sep 09 06:52:25.695454 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:25.695435    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfdk5" event={"ID":"20f229be-0318-40fb-aa2a-62e0df181580","Type":"ContainerStarted","Data":"a96a14314ecb833a5cb9010ef2b5a288ec2d5868ab27c79e0d033076e993020d"}
Sep 09 06:52:25.737931 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:25.737549    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mfdk5" podStartSLOduration=67.149358918 podStartE2EDuration="1m8.73752533s" totalImagesPullingTime="1.588166412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:25.736083672 +0000 UTC m=+68.956324786" watchObservedRunningTime="2026-09-09 06:52:25.73752533 +0000 UTC m=+68.957766445"
Sep 09 06:52:26.040249 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:52:26.040168    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2094e263_ae7e_437d_ad57_2bc058bb72bd.slice/crio-da79813785056cd08a09d650fc238a7ecec1678e300601e65e828ee9317d2c82 WatchSource:0}: Error finding container da79813785056cd08a09d650fc238a7ecec1678e300601e65e828ee9317d2c82: Status 404 returned error can't find the container with id da79813785056cd08a09d650fc238a7ecec1678e300601e65e828ee9317d2c82
Sep 09 06:52:26.701658 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:26.701617    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w" event={"ID":"2094e263-ae7e-437d-ad57-2bc058bb72bd","Type":"ContainerStarted","Data":"da79813785056cd08a09d650fc238a7ecec1678e300601e65e828ee9317d2c82"}
Sep 09 06:52:29.716636 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.716515    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fct4f" event={"ID":"2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d","Type":"ContainerStarted","Data":"22aaf752a7ca61de7354de699f143665af9a06d83444ba6843f48b24116bd994"}
Sep 09 06:52:29.719724 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.719633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rj9m" event={"ID":"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082","Type":"ContainerStarted","Data":"d8a848650b3bf4450925167ca0172e963b2e56f70f5c6d5d2836080c8cc0478b"}
Sep 09 06:52:29.719724 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.719687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rj9m" event={"ID":"8f4ffda3-2dfb-47c8-b650-e7a7f7adc082","Type":"ContainerStarted","Data":"d0d53f64d6d0fefcff805d6df6ce84da6281c0b09f1282c6473f8e73ca761fb9"}
Sep 09 06:52:29.719724 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.719716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:52:29.721867 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.721512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjl5t" event={"ID":"9552e261-a17b-4376-99b3-5ff9f8f87b0e","Type":"ContainerStarted","Data":"aac0f089bb2b7774ed16995e6b1302dc0c6453430978cac6f8d878646846f383"}
Sep 09 06:52:29.721867 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.721737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:52:29.734610 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.734555    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fct4f" podStartSLOduration=66.805625696 podStartE2EDuration="1m12.734536332s" totalImagesPullingTime="5.928910636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:29.732888529 +0000 UTC m=+72.953129642" watchObservedRunningTime="2026-09-09 06:52:29.734536332 +0000 UTC m=+72.954777441"
Sep 09 06:52:29.750679 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.750626    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kjl5t" podStartSLOduration=67.124685098 podStartE2EDuration="1m12.750609702s" totalImagesPullingTime="5.625924604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:29.749894388 +0000 UTC m=+72.970135502" watchObservedRunningTime="2026-09-09 06:52:29.750609702 +0000 UTC m=+72.970850815"
Sep 09 06:52:29.768616 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:29.768565    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7rj9m" podStartSLOduration=33.104501809 podStartE2EDuration="38.768545607s" totalImagesPullingTime="5.664043798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:29.767623729 +0000 UTC m=+72.987864843" watchObservedRunningTime="2026-09-09 06:52:29.768545607 +0000 UTC m=+72.988786721"
Sep 09 06:52:32.733829 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:32.733790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w" event={"ID":"2094e263-ae7e-437d-ad57-2bc058bb72bd","Type":"ContainerStarted","Data":"a515b75494d8866f2ebd853fcbdad40fc3648a5f2626e896aa1ed34dbb7c7733"}
Sep 09 06:52:32.734244 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:32.733838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w" event={"ID":"2094e263-ae7e-437d-ad57-2bc058bb72bd","Type":"ContainerStarted","Data":"d3c106c2fa8209f48f93b3112b08cb07eff2c67a7353f3346a3c5166efa3fa72"}
Sep 09 06:52:32.734244 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:32.733853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w" event={"ID":"2094e263-ae7e-437d-ad57-2bc058bb72bd","Type":"ContainerStarted","Data":"19ea70188939a951fc721c766148d7bf008bd408b3cf9c456da154d6953300f4"}
Sep 09 06:52:32.754400 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:32.754355    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65d75c7f68-lsb2w" podStartSLOduration=2.765079666 podStartE2EDuration="8.754340505s" totalImagesPullingTime="5.989260839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:32.75261843 +0000 UTC m=+75.972859545" watchObservedRunningTime="2026-09-09 06:52:32.754340505 +0000 UTC m=+75.974581652"
Sep 09 06:52:34.869960 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:34.869931    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:52:37.829720 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.829681    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v4zhb"]
Sep 09 06:52:37.832983 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.832958    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.835559 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.835536    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 06:52:37.835795 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.835771    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 06:52:37.835906 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.835820    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-97pmt\""
Sep 09 06:52:37.835906 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.835882    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 06:52:37.837096 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.837077    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 06:52:37.837247 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.837137    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 06:52:37.837247 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.837210    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 06:52:37.936163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.936163 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-accelerators-collector-config\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.936396 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-tls\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.936396 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-wtmp\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.936396 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-textfile\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.936396 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-sys\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.936586 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb1bd352-477f-44c7-99fc-3706c9fd43df-metrics-client-ca\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.936586 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrj2\" (UniqueName: \"kubernetes.io/projected/cb1bd352-477f-44c7-99fc-3706c9fd43df-kube-api-access-5lrj2\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:37.936586 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:37.936500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-root\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037404 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037580 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-accelerators-collector-config\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037580 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-tls\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037580 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-wtmp\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037580 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-textfile\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037783 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-sys\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037783 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037620    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-wtmp\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037783 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb1bd352-477f-44c7-99fc-3706c9fd43df-metrics-client-ca\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037783 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrj2\" (UniqueName: \"kubernetes.io/projected/cb1bd352-477f-44c7-99fc-3706c9fd43df-kube-api-access-5lrj2\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037783 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-root\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.037783 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-sys\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.038066 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-textfile\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.038066 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.037841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cb1bd352-477f-44c7-99fc-3706c9fd43df-root\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.038205 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.038157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-accelerators-collector-config\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.038310 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.038292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb1bd352-477f-44c7-99fc-3706c9fd43df-metrics-client-ca\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.039763 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.039743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-tls\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.039859 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.039841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb1bd352-477f-44c7-99fc-3706c9fd43df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.048555 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.048535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrj2\" (UniqueName: \"kubernetes.io/projected/cb1bd352-477f-44c7-99fc-3706c9fd43df-kube-api-access-5lrj2\") pod \"node-exporter-v4zhb\" (UID: \"cb1bd352-477f-44c7-99fc-3706c9fd43df\") " pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.142640 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.142566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v4zhb"
Sep 09 06:52:38.153696 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:52:38.153666    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1bd352_477f_44c7_99fc_3706c9fd43df.slice/crio-1919cab9a09ce7e3839b10c3dfed2ce637af3f581239c5152e0a26aa478f4bb3 WatchSource:0}: Error finding container 1919cab9a09ce7e3839b10c3dfed2ce637af3f581239c5152e0a26aa478f4bb3: Status 404 returned error can't find the container with id 1919cab9a09ce7e3839b10c3dfed2ce637af3f581239c5152e0a26aa478f4bb3
Sep 09 06:52:38.758247 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:38.758206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v4zhb" event={"ID":"cb1bd352-477f-44c7-99fc-3706c9fd43df","Type":"ContainerStarted","Data":"1919cab9a09ce7e3839b10c3dfed2ce637af3f581239c5152e0a26aa478f4bb3"}
Sep 09 06:52:39.727151 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:39.727120    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7rj9m"
Sep 09 06:52:39.763091 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:39.763056    2562 generic.go:356] "Generic (PLEG): container finished" podID="cb1bd352-477f-44c7-99fc-3706c9fd43df" containerID="7cdb6559da32f1d6d8c8a2330d2bd0e98df77ad9fe286e1b9b10cc1e5f46da66" exitCode=0
Sep 09 06:52:39.763271 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:39.763135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v4zhb" event={"ID":"cb1bd352-477f-44c7-99fc-3706c9fd43df","Type":"ContainerDied","Data":"7cdb6559da32f1d6d8c8a2330d2bd0e98df77ad9fe286e1b9b10cc1e5f46da66"}
Sep 09 06:52:40.768732 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:40.768696    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v4zhb" event={"ID":"cb1bd352-477f-44c7-99fc-3706c9fd43df","Type":"ContainerStarted","Data":"1eceee9bcdd1abb57e07878476905c92671f4678601f4bf6b8521c5c7b77bcf0"}
Sep 09 06:52:40.769153 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:40.768745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v4zhb" event={"ID":"cb1bd352-477f-44c7-99fc-3706c9fd43df","Type":"ContainerStarted","Data":"8a60217333d6fc90f3fdbc06178d2c22b5a2b00a7449a94f8427b2dc9078014c"}
Sep 09 06:52:40.789803 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:40.789753    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v4zhb" podStartSLOduration=2.705122008 podStartE2EDuration="3.789737908s" totalImagesPullingTime="1.023530376s" totalInitContainerRuntime="61.085524ms" isStatefulPod=true podCreationTimestamp="2026-09-09 06:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:52:40.78819892 +0000 UTC m=+84.008440034" watchObservedRunningTime="2026-09-09 06:52:40.789737908 +0000 UTC m=+84.009979020"
Sep 09 06:52:49.900847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:49.900797    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt" podUID="fa15c0ca-032c-4568-ac2f-94a73a870ddf" containerName="registry" containerID="cri-o://cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890" gracePeriod=30
Sep 09 06:52:50.127215 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.127191    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:52:50.230200 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230165    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa15c0ca-032c-4568-ac2f-94a73a870ddf-ca-trust-extracted\") pod \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") "
Sep 09 06:52:50.230200 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230204    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-certificates\") pod \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") "
Sep 09 06:52:50.230453 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230244    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") pod \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") "
Sep 09 06:52:50.230453 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230266    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-installation-pull-secrets\") pod \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") "
Sep 09 06:52:50.230453 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230283    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-bound-sa-token\") pod \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") "
Sep 09 06:52:50.230453 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230319    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-trusted-ca\") pod \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") "
Sep 09 06:52:50.230453 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230338    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-image-registry-private-configuration\") pod \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") "
Sep 09 06:52:50.230453 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230359    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7lhl\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-kube-api-access-f7lhl\") pod \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\" (UID: \"fa15c0ca-032c-4568-ac2f-94a73a870ddf\") "
Sep 09 06:52:50.230782 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230614    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fa15c0ca-032c-4568-ac2f-94a73a870ddf" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:52:50.230904 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.230879    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fa15c0ca-032c-4568-ac2f-94a73a870ddf" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 06:52:50.232977 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.232947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-kube-api-access-f7lhl" (OuterVolumeSpecName: "kube-api-access-f7lhl") pod "fa15c0ca-032c-4568-ac2f-94a73a870ddf" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf"). InnerVolumeSpecName "kube-api-access-f7lhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:52:50.233160 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.232986    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fa15c0ca-032c-4568-ac2f-94a73a870ddf" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:52:50.233160 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.233023    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fa15c0ca-032c-4568-ac2f-94a73a870ddf" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:52:50.233160 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.233069    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fa15c0ca-032c-4568-ac2f-94a73a870ddf" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 06:52:50.233160 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.233097    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fa15c0ca-032c-4568-ac2f-94a73a870ddf" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:52:50.239224 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.239201    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa15c0ca-032c-4568-ac2f-94a73a870ddf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fa15c0ca-032c-4568-ac2f-94a73a870ddf" (UID: "fa15c0ca-032c-4568-ac2f-94a73a870ddf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:52:50.331098 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.331063    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-certificates\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:52:50.331098 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.331093    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-registry-tls\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:52:50.331098 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.331120    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-installation-pull-secrets\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:52:50.331320 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.331131    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-bound-sa-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:52:50.331320 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.331141    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0ca-032c-4568-ac2f-94a73a870ddf-trusted-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:52:50.331320 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.331149    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa15c0ca-032c-4568-ac2f-94a73a870ddf-image-registry-private-configuration\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:52:50.331320 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.331157    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7lhl\" (UniqueName: \"kubernetes.io/projected/fa15c0ca-032c-4568-ac2f-94a73a870ddf-kube-api-access-f7lhl\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:52:50.331320 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.331167    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa15c0ca-032c-4568-ac2f-94a73a870ddf-ca-trust-extracted\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:52:50.806481 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.806443    2562 generic.go:356] "Generic (PLEG): container finished" podID="fa15c0ca-032c-4568-ac2f-94a73a870ddf" containerID="cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890" exitCode=0
Sep 09 06:52:50.806652 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.806486    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt"
Sep 09 06:52:50.806652 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.806487    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt" event={"ID":"fa15c0ca-032c-4568-ac2f-94a73a870ddf","Type":"ContainerDied","Data":"cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890"}
Sep 09 06:52:50.806652 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.806614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b5bc69fb-rlpgt" event={"ID":"fa15c0ca-032c-4568-ac2f-94a73a870ddf","Type":"ContainerDied","Data":"f33046970ea6d436c08d63c18f37e1367121d5e75f06e6e519560ba3df6c7c4c"}
Sep 09 06:52:50.807462 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.807446    2562 scope.go:117] "RemoveContainer" containerID="cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890"
Sep 09 06:52:50.816576 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.816561    2562 scope.go:117] "RemoveContainer" containerID="cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890"
Sep 09 06:52:50.816829 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:52:50.816809    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890\": container with ID starting with cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890 not found: ID does not exist" containerID="cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890"
Sep 09 06:52:50.816888 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.816836    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890"} err="failed to get container status \"cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890\": rpc error: code = NotFound desc = could not find container \"cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890\": container with ID starting with cc9f214f235b2836eca565cbfbb6d753bb32e3215653fc8b68d32dadff177890 not found: ID does not exist"
Sep 09 06:52:50.833545 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.833521    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64b5bc69fb-rlpgt"]
Sep 09 06:52:50.841908 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:50.841885    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64b5bc69fb-rlpgt"]
Sep 09 06:52:51.228537 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:52:51.228504    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa15c0ca-032c-4568-ac2f-94a73a870ddf" path="/var/lib/kubelet/pods/fa15c0ca-032c-4568-ac2f-94a73a870ddf/volumes"
Sep 09 06:53:00.727473 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:00.727443    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kjl5t"
Sep 09 06:53:23.914417 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:23.914382    2562 generic.go:356] "Generic (PLEG): container finished" podID="bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6" containerID="e8b47a4edb069a66d21126442519be9367f0755dd3b5a4857ea6275a9f446af4" exitCode=0
Sep 09 06:53:23.914811 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:23.914455    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2" event={"ID":"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6","Type":"ContainerDied","Data":"e8b47a4edb069a66d21126442519be9367f0755dd3b5a4857ea6275a9f446af4"}
Sep 09 06:53:23.914811 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:23.914748    2562 scope.go:117] "RemoveContainer" containerID="e8b47a4edb069a66d21126442519be9367f0755dd3b5a4857ea6275a9f446af4"
Sep 09 06:53:24.920825 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:24.920793    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qc9k2" event={"ID":"bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6","Type":"ContainerStarted","Data":"72e15385019e84eebfba262a078fce18dac39952dcff68d9e9e56a55e5b43192"}
Sep 09 06:53:24.922271 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:24.922247    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e13bfba-38d5-4775-9a82-b74c70758943" containerID="2310dc59e87491c114ad78fe8d32c9079d892b5597061dfa88d8efba80051507" exitCode=0
Sep 09 06:53:24.922387 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:24.922299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj" event={"ID":"5e13bfba-38d5-4775-9a82-b74c70758943","Type":"ContainerDied","Data":"2310dc59e87491c114ad78fe8d32c9079d892b5597061dfa88d8efba80051507"}
Sep 09 06:53:24.922554 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:24.922541    2562 scope.go:117] "RemoveContainer" containerID="2310dc59e87491c114ad78fe8d32c9079d892b5597061dfa88d8efba80051507"
Sep 09 06:53:25.927542 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:53:25.927508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-52mjj" event={"ID":"5e13bfba-38d5-4775-9a82-b74c70758943","Type":"ContainerStarted","Data":"ba248875622e45d25bbfac102b9005a04af4e37efeb4fe58cf4c078221dd788b"}
Sep 09 06:55:31.992196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:31.992166    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"]
Sep 09 06:55:31.992796 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:31.992432    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa15c0ca-032c-4568-ac2f-94a73a870ddf" containerName="registry"
Sep 09 06:55:31.992796 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:31.992445    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa15c0ca-032c-4568-ac2f-94a73a870ddf" containerName="registry"
Sep 09 06:55:31.992796 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:31.992499    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa15c0ca-032c-4568-ac2f-94a73a870ddf" containerName="registry"
Sep 09 06:55:31.994261 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:31.994245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:31.996556 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:31.996534    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phbjw\""
Sep 09 06:55:31.996556 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:31.996543    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 06:55:31.997279 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:31.997257    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 06:55:32.004224 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.004200    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"]
Sep 09 06:55:32.105183 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.105145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k5c7\" (UniqueName: \"kubernetes.io/projected/f9b3ecff-3d0c-4fae-a2e7-612116154c80-kube-api-access-7k5c7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.105183 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.105181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.105402 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.105203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.206282 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.206253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k5c7\" (UniqueName: \"kubernetes.io/projected/f9b3ecff-3d0c-4fae-a2e7-612116154c80-kube-api-access-7k5c7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.206282 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.206287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.206469 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.206305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.206718 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.206698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.206718 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.206710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.216969 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.216943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k5c7\" (UniqueName: \"kubernetes.io/projected/f9b3ecff-3d0c-4fae-a2e7-612116154c80-kube-api-access-7k5c7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.303269 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.303195    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:32.572146 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:32.572062    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"]
Sep 09 06:55:32.574890 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:55:32.574866    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b3ecff_3d0c_4fae_a2e7_612116154c80.slice/crio-47679efabc03e251353a52384763f4fa8744be622e7f4c8bda70e5e9176594e5 WatchSource:0}: Error finding container 47679efabc03e251353a52384763f4fa8744be622e7f4c8bda70e5e9176594e5: Status 404 returned error can't find the container with id 47679efabc03e251353a52384763f4fa8744be622e7f4c8bda70e5e9176594e5
Sep 09 06:55:33.340431 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:33.340398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8" event={"ID":"f9b3ecff-3d0c-4fae-a2e7-612116154c80","Type":"ContainerStarted","Data":"47679efabc03e251353a52384763f4fa8744be622e7f4c8bda70e5e9176594e5"}
Sep 09 06:55:39.363281 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:39.363245    2562 generic.go:356] "Generic (PLEG): container finished" podID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerID="384cabd1b5ea5cf8777b73d9c957157cb2a9c446806d08a46efa736e9f583f2a" exitCode=0
Sep 09 06:55:39.363281 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:39.363287    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8" event={"ID":"f9b3ecff-3d0c-4fae-a2e7-612116154c80","Type":"ContainerDied","Data":"384cabd1b5ea5cf8777b73d9c957157cb2a9c446806d08a46efa736e9f583f2a"}
Sep 09 06:55:41.372485 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:41.372453    2562 generic.go:356] "Generic (PLEG): container finished" podID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerID="3f747eff00ed891b6fef95eb73f8801f49e6147e3609a1d6aa9d8c72707ab05c" exitCode=0
Sep 09 06:55:41.372820 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:41.372538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8" event={"ID":"f9b3ecff-3d0c-4fae-a2e7-612116154c80","Type":"ContainerDied","Data":"3f747eff00ed891b6fef95eb73f8801f49e6147e3609a1d6aa9d8c72707ab05c"}
Sep 09 06:55:48.401288 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:48.401244    2562 generic.go:356] "Generic (PLEG): container finished" podID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerID="581943518c406882562f91c87ae0b5c33f2b99e11d5730a0d217153be605ee94" exitCode=0
Sep 09 06:55:48.401665 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:48.401340    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8" event={"ID":"f9b3ecff-3d0c-4fae-a2e7-612116154c80","Type":"ContainerDied","Data":"581943518c406882562f91c87ae0b5c33f2b99e11d5730a0d217153be605ee94"}
Sep 09 06:55:49.523390 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.523368    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:49.647264 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.647235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k5c7\" (UniqueName: \"kubernetes.io/projected/f9b3ecff-3d0c-4fae-a2e7-612116154c80-kube-api-access-7k5c7\") pod \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") "
Sep 09 06:55:49.647264 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.647267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-bundle\") pod \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") "
Sep 09 06:55:49.647473 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.647299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-util\") pod \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\" (UID: \"f9b3ecff-3d0c-4fae-a2e7-612116154c80\") "
Sep 09 06:55:49.647972 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.647946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-bundle" (OuterVolumeSpecName: "bundle") pod "f9b3ecff-3d0c-4fae-a2e7-612116154c80" (UID: "f9b3ecff-3d0c-4fae-a2e7-612116154c80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:55:49.649454 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.649422    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b3ecff-3d0c-4fae-a2e7-612116154c80-kube-api-access-7k5c7" (OuterVolumeSpecName: "kube-api-access-7k5c7") pod "f9b3ecff-3d0c-4fae-a2e7-612116154c80" (UID: "f9b3ecff-3d0c-4fae-a2e7-612116154c80"). InnerVolumeSpecName "kube-api-access-7k5c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:55:49.652148 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.652117    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-util" (OuterVolumeSpecName: "util") pod "f9b3ecff-3d0c-4fae-a2e7-612116154c80" (UID: "f9b3ecff-3d0c-4fae-a2e7-612116154c80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:55:49.748168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.748126    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k5c7\" (UniqueName: \"kubernetes.io/projected/f9b3ecff-3d0c-4fae-a2e7-612116154c80-kube-api-access-7k5c7\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:55:49.748168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.748163    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-bundle\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:55:49.748168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:49.748172    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b3ecff-3d0c-4fae-a2e7-612116154c80-util\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:55:50.410238 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:50.410207    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8"
Sep 09 06:55:50.410433 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:50.410210    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540f2ns8" event={"ID":"f9b3ecff-3d0c-4fae-a2e7-612116154c80","Type":"ContainerDied","Data":"47679efabc03e251353a52384763f4fa8744be622e7f4c8bda70e5e9176594e5"}
Sep 09 06:55:50.410433 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:50.410322    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47679efabc03e251353a52384763f4fa8744be622e7f4c8bda70e5e9176594e5"
Sep 09 06:55:55.201985 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.201951    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"]
Sep 09 06:55:55.202369 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.202192    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerName="pull"
Sep 09 06:55:55.202369 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.202203    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerName="pull"
Sep 09 06:55:55.202369 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.202213    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerName="extract"
Sep 09 06:55:55.202369 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.202220    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerName="extract"
Sep 09 06:55:55.202369 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.202227    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerName="util"
Sep 09 06:55:55.202369 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.202233    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerName="util"
Sep 09 06:55:55.202369 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.202273    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b3ecff-3d0c-4fae-a2e7-612116154c80" containerName="extract"
Sep 09 06:55:55.246217 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.246184    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"]
Sep 09 06:55:55.246353 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.246311    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"
Sep 09 06:55:55.249032 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.249009    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p59t7\""
Sep 09 06:55:55.249196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.249013    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 06:55:55.249196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.249074    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 06:55:55.283363 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.283338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxlrx\" (UniqueName: \"kubernetes.io/projected/3b2131f2-6850-40dd-8a6c-366226c103f3-kube-api-access-gxlrx\") pod \"cert-manager-operator-controller-manager-55c444bf86-vl8bf\" (UID: \"3b2131f2-6850-40dd-8a6c-366226c103f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"
Sep 09 06:55:55.283463 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.283373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b2131f2-6850-40dd-8a6c-366226c103f3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vl8bf\" (UID: \"3b2131f2-6850-40dd-8a6c-366226c103f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"
Sep 09 06:55:55.384324 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.384285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b2131f2-6850-40dd-8a6c-366226c103f3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vl8bf\" (UID: \"3b2131f2-6850-40dd-8a6c-366226c103f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"
Sep 09 06:55:55.384471 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.384347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxlrx\" (UniqueName: \"kubernetes.io/projected/3b2131f2-6850-40dd-8a6c-366226c103f3-kube-api-access-gxlrx\") pod \"cert-manager-operator-controller-manager-55c444bf86-vl8bf\" (UID: \"3b2131f2-6850-40dd-8a6c-366226c103f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"
Sep 09 06:55:55.384704 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.384681    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b2131f2-6850-40dd-8a6c-366226c103f3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vl8bf\" (UID: \"3b2131f2-6850-40dd-8a6c-366226c103f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"
Sep 09 06:55:55.394094 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.394067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxlrx\" (UniqueName: \"kubernetes.io/projected/3b2131f2-6850-40dd-8a6c-366226c103f3-kube-api-access-gxlrx\") pod \"cert-manager-operator-controller-manager-55c444bf86-vl8bf\" (UID: \"3b2131f2-6850-40dd-8a6c-366226c103f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"
Sep 09 06:55:55.555229 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.555151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"
Sep 09 06:55:55.850330 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:55.850247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf"]
Sep 09 06:55:55.853285 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:55:55.853256    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2131f2_6850_40dd_8a6c_366226c103f3.slice/crio-982cec1bc4a62482d504e540fa2558ea41de2e749c7cb7e5b99930d4463971b3 WatchSource:0}: Error finding container 982cec1bc4a62482d504e540fa2558ea41de2e749c7cb7e5b99930d4463971b3: Status 404 returned error can't find the container with id 982cec1bc4a62482d504e540fa2558ea41de2e749c7cb7e5b99930d4463971b3
Sep 09 06:55:56.432023 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:56.431987    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf" event={"ID":"3b2131f2-6850-40dd-8a6c-366226c103f3","Type":"ContainerStarted","Data":"982cec1bc4a62482d504e540fa2558ea41de2e749c7cb7e5b99930d4463971b3"}
Sep 09 06:55:58.441326 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:58.441288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf" event={"ID":"3b2131f2-6850-40dd-8a6c-366226c103f3","Type":"ContainerStarted","Data":"620ab0a2e54beb75c19ee295057bdd1702d0d25e861327840b004dec6a51aa45"}
Sep 09 06:55:58.468681 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:58.468622    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vl8bf" podStartSLOduration=1.549747537 podStartE2EDuration="3.468603984s" totalImagesPullingTime="1.918856447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:55:58.467297562 +0000 UTC m=+281.687538678" watchObservedRunningTime="2026-09-09 06:55:58.468603984 +0000 UTC m=+281.688845101"
Sep 09 06:55:59.658997 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.658966    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"]
Sep 09 06:55:59.683822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.683792    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"]
Sep 09 06:55:59.683960 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.683910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.687167 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.687143    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phbjw\""
Sep 09 06:55:59.687291 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.687229    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 06:55:59.687924 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.687907    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 06:55:59.773589 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.773554    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f2bpv"]
Sep 09 06:55:59.795748 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.795716    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f2bpv"]
Sep 09 06:55:59.795883 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.795805    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:55:59.798319 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.798300    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 06:55:59.798548 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.798523    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 06:55:59.798647 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.798549    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xqj2j\""
Sep 09 06:55:59.816961 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.816933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl48l\" (UniqueName: \"kubernetes.io/projected/a8a34134-f4be-4588-be1f-34f2e038e5bb-kube-api-access-nl48l\") pod \"cert-manager-webhook-6486ffd997-f2bpv\" (UID: \"a8a34134-f4be-4588-be1f-34f2e038e5bb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:55:59.817057 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.816977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.817057 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.817003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rnp\" (UniqueName: \"kubernetes.io/projected/4835ac17-9f6c-421f-a40c-e6b262a05c15-kube-api-access-86rnp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.817150 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.817074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8a34134-f4be-4588-be1f-34f2e038e5bb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f2bpv\" (UID: \"a8a34134-f4be-4588-be1f-34f2e038e5bb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:55:59.817150 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.817136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.917533 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.917503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.917653 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.917556    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl48l\" (UniqueName: \"kubernetes.io/projected/a8a34134-f4be-4588-be1f-34f2e038e5bb-kube-api-access-nl48l\") pod \"cert-manager-webhook-6486ffd997-f2bpv\" (UID: \"a8a34134-f4be-4588-be1f-34f2e038e5bb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:55:59.917653 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.917586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.917653 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.917602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rnp\" (UniqueName: \"kubernetes.io/projected/4835ac17-9f6c-421f-a40c-e6b262a05c15-kube-api-access-86rnp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.917653 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.917632    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8a34134-f4be-4588-be1f-34f2e038e5bb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f2bpv\" (UID: \"a8a34134-f4be-4588-be1f-34f2e038e5bb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:55:59.917869 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.917846    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.917932 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.917919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.929250 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.929216    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8a34134-f4be-4588-be1f-34f2e038e5bb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f2bpv\" (UID: \"a8a34134-f4be-4588-be1f-34f2e038e5bb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:55:59.929878 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.929861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl48l\" (UniqueName: \"kubernetes.io/projected/a8a34134-f4be-4588-be1f-34f2e038e5bb-kube-api-access-nl48l\") pod \"cert-manager-webhook-6486ffd997-f2bpv\" (UID: \"a8a34134-f4be-4588-be1f-34f2e038e5bb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:55:59.930151 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.930130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rnp\" (UniqueName: \"kubernetes.io/projected/4835ac17-9f6c-421f-a40c-e6b262a05c15-kube-api-access-86rnp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:55:59.993242 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:55:59.993214    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:56:00.104577 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:00.104537    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:56:00.275021 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:00.274990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"]
Sep 09 06:56:00.278762 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:56:00.278733    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4835ac17_9f6c_421f_a40c_e6b262a05c15.slice/crio-770a698686a55eee6b3b8c7f4eba316ca528b74c323cfdf73a3dc85f4549fd7a WatchSource:0}: Error finding container 770a698686a55eee6b3b8c7f4eba316ca528b74c323cfdf73a3dc85f4549fd7a: Status 404 returned error can't find the container with id 770a698686a55eee6b3b8c7f4eba316ca528b74c323cfdf73a3dc85f4549fd7a
Sep 09 06:56:00.397006 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:00.396974    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f2bpv"]
Sep 09 06:56:00.400355 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:56:00.400324    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8a34134_f4be_4588_be1f_34f2e038e5bb.slice/crio-34c7061549fb9659254830df5d345e59416c0d31f50293cba3dc304725bad2e2 WatchSource:0}: Error finding container 34c7061549fb9659254830df5d345e59416c0d31f50293cba3dc304725bad2e2: Status 404 returned error can't find the container with id 34c7061549fb9659254830df5d345e59416c0d31f50293cba3dc304725bad2e2
Sep 09 06:56:00.449804 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:00.449777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv" event={"ID":"a8a34134-f4be-4588-be1f-34f2e038e5bb","Type":"ContainerStarted","Data":"34c7061549fb9659254830df5d345e59416c0d31f50293cba3dc304725bad2e2"}
Sep 09 06:56:00.451134 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:00.451087    2562 generic.go:356] "Generic (PLEG): container finished" podID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerID="d920dcd5ded3a0d21949d31adf4bad12fdfec73f53e6093bfc52ebc8f84de640" exitCode=0
Sep 09 06:56:00.451224 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:00.451133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6" event={"ID":"4835ac17-9f6c-421f-a40c-e6b262a05c15","Type":"ContainerDied","Data":"d920dcd5ded3a0d21949d31adf4bad12fdfec73f53e6093bfc52ebc8f84de640"}
Sep 09 06:56:00.451224 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:00.451176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6" event={"ID":"4835ac17-9f6c-421f-a40c-e6b262a05c15","Type":"ContainerStarted","Data":"770a698686a55eee6b3b8c7f4eba316ca528b74c323cfdf73a3dc85f4549fd7a"}
Sep 09 06:56:04.470287 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:04.470249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv" event={"ID":"a8a34134-f4be-4588-be1f-34f2e038e5bb","Type":"ContainerStarted","Data":"e2243e574f39a6838a9fa6e21a5a746387dd1a01fcb2d9514eadddcc1ae140e5"}
Sep 09 06:56:04.470714 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:04.470299    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:56:04.472142 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:04.472116    2562 generic.go:356] "Generic (PLEG): container finished" podID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerID="f75b493d876fea4b154b8916514232e08c1777e0f9883fdadc35ac6f9b9a9eea" exitCode=0
Sep 09 06:56:04.472254 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:04.472148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6" event={"ID":"4835ac17-9f6c-421f-a40c-e6b262a05c15","Type":"ContainerDied","Data":"f75b493d876fea4b154b8916514232e08c1777e0f9883fdadc35ac6f9b9a9eea"}
Sep 09 06:56:04.487882 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:04.487744    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv" podStartSLOduration=2.42239132 podStartE2EDuration="5.487726543s" totalImagesPullingTime="3.065335223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:56:04.487606845 +0000 UTC m=+287.707847958" watchObservedRunningTime="2026-09-09 06:56:04.487726543 +0000 UTC m=+287.707967656"
Sep 09 06:56:05.478524 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:05.478489    2562 generic.go:356] "Generic (PLEG): container finished" podID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerID="5f73751df4f9377b5696e25a77dfaffee82b30b29a4798b6fc60d58dd76c001b" exitCode=0
Sep 09 06:56:05.478969 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:05.478564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6" event={"ID":"4835ac17-9f6c-421f-a40c-e6b262a05c15","Type":"ContainerDied","Data":"5f73751df4f9377b5696e25a77dfaffee82b30b29a4798b6fc60d58dd76c001b"}
Sep 09 06:56:06.602006 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.601983    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:56:06.666117 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.666085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86rnp\" (UniqueName: \"kubernetes.io/projected/4835ac17-9f6c-421f-a40c-e6b262a05c15-kube-api-access-86rnp\") pod \"4835ac17-9f6c-421f-a40c-e6b262a05c15\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") "
Sep 09 06:56:06.666282 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.666163    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-bundle\") pod \"4835ac17-9f6c-421f-a40c-e6b262a05c15\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") "
Sep 09 06:56:06.666282 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.666187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-util\") pod \"4835ac17-9f6c-421f-a40c-e6b262a05c15\" (UID: \"4835ac17-9f6c-421f-a40c-e6b262a05c15\") "
Sep 09 06:56:06.666561 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.666530    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-bundle" (OuterVolumeSpecName: "bundle") pod "4835ac17-9f6c-421f-a40c-e6b262a05c15" (UID: "4835ac17-9f6c-421f-a40c-e6b262a05c15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:56:06.668183 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.668151    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4835ac17-9f6c-421f-a40c-e6b262a05c15-kube-api-access-86rnp" (OuterVolumeSpecName: "kube-api-access-86rnp") pod "4835ac17-9f6c-421f-a40c-e6b262a05c15" (UID: "4835ac17-9f6c-421f-a40c-e6b262a05c15"). InnerVolumeSpecName "kube-api-access-86rnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:56:06.670118 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.670083    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-util" (OuterVolumeSpecName: "util") pod "4835ac17-9f6c-421f-a40c-e6b262a05c15" (UID: "4835ac17-9f6c-421f-a40c-e6b262a05c15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:56:06.766868 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.766788    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86rnp\" (UniqueName: \"kubernetes.io/projected/4835ac17-9f6c-421f-a40c-e6b262a05c15-kube-api-access-86rnp\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:06.766868 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.766817    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-bundle\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:06.766868 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:06.766827    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4835ac17-9f6c-421f-a40c-e6b262a05c15-util\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:07.488055 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:07.487967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6" event={"ID":"4835ac17-9f6c-421f-a40c-e6b262a05c15","Type":"ContainerDied","Data":"770a698686a55eee6b3b8c7f4eba316ca528b74c323cfdf73a3dc85f4549fd7a"}
Sep 09 06:56:07.488055 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:07.488023    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="770a698686a55eee6b3b8c7f4eba316ca528b74c323cfdf73a3dc85f4549fd7a"
Sep 09 06:56:07.488055 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:07.487984    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnq4h6"
Sep 09 06:56:10.480510 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:10.480484    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f2bpv"
Sep 09 06:56:11.391049 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.391015    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dq8sm"]
Sep 09 06:56:11.391260 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.391247    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerName="extract"
Sep 09 06:56:11.391308 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.391261    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerName="extract"
Sep 09 06:56:11.391308 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.391278    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerName="util"
Sep 09 06:56:11.391308 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.391283    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerName="util"
Sep 09 06:56:11.391308 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.391290    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerName="pull"
Sep 09 06:56:11.391308 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.391295    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerName="pull"
Sep 09 06:56:11.391447 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.391345    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4835ac17-9f6c-421f-a40c-e6b262a05c15" containerName="extract"
Sep 09 06:56:11.392840 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.392824    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dq8sm"
Sep 09 06:56:11.395245 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.395222    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9qdp7\""
Sep 09 06:56:11.404062 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.404039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dq8sm"]
Sep 09 06:56:11.498461 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.498428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqhtx\" (UniqueName: \"kubernetes.io/projected/9339c265-50ca-4f68-9bef-a6a95a86cb88-kube-api-access-mqhtx\") pod \"cert-manager-55b4d954d5-dq8sm\" (UID: \"9339c265-50ca-4f68-9bef-a6a95a86cb88\") " pod="cert-manager/cert-manager-55b4d954d5-dq8sm"
Sep 09 06:56:11.498847 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.498473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9339c265-50ca-4f68-9bef-a6a95a86cb88-bound-sa-token\") pod \"cert-manager-55b4d954d5-dq8sm\" (UID: \"9339c265-50ca-4f68-9bef-a6a95a86cb88\") " pod="cert-manager/cert-manager-55b4d954d5-dq8sm"
Sep 09 06:56:11.599203 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.599165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqhtx\" (UniqueName: \"kubernetes.io/projected/9339c265-50ca-4f68-9bef-a6a95a86cb88-kube-api-access-mqhtx\") pod \"cert-manager-55b4d954d5-dq8sm\" (UID: \"9339c265-50ca-4f68-9bef-a6a95a86cb88\") " pod="cert-manager/cert-manager-55b4d954d5-dq8sm"
Sep 09 06:56:11.599368 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.599226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9339c265-50ca-4f68-9bef-a6a95a86cb88-bound-sa-token\") pod \"cert-manager-55b4d954d5-dq8sm\" (UID: \"9339c265-50ca-4f68-9bef-a6a95a86cb88\") " pod="cert-manager/cert-manager-55b4d954d5-dq8sm"
Sep 09 06:56:11.609241 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.609205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9339c265-50ca-4f68-9bef-a6a95a86cb88-bound-sa-token\") pod \"cert-manager-55b4d954d5-dq8sm\" (UID: \"9339c265-50ca-4f68-9bef-a6a95a86cb88\") " pod="cert-manager/cert-manager-55b4d954d5-dq8sm"
Sep 09 06:56:11.609457 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.609437    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqhtx\" (UniqueName: \"kubernetes.io/projected/9339c265-50ca-4f68-9bef-a6a95a86cb88-kube-api-access-mqhtx\") pod \"cert-manager-55b4d954d5-dq8sm\" (UID: \"9339c265-50ca-4f68-9bef-a6a95a86cb88\") " pod="cert-manager/cert-manager-55b4d954d5-dq8sm"
Sep 09 06:56:11.701947 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.701918    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dq8sm"
Sep 09 06:56:11.969937 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:11.969891    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dq8sm"]
Sep 09 06:56:11.972486 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:56:11.972459    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9339c265_50ca_4f68_9bef_a6a95a86cb88.slice/crio-29b2bbceb71524f227511dbcc901b7fe15c6abad40b8983b35805b312b6f1ac7 WatchSource:0}: Error finding container 29b2bbceb71524f227511dbcc901b7fe15c6abad40b8983b35805b312b6f1ac7: Status 404 returned error can't find the container with id 29b2bbceb71524f227511dbcc901b7fe15c6abad40b8983b35805b312b6f1ac7
Sep 09 06:56:12.514046 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:12.514011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dq8sm" event={"ID":"9339c265-50ca-4f68-9bef-a6a95a86cb88","Type":"ContainerStarted","Data":"ac532e5fa3fcae778e5e170c5f8702166211b0fa2a6d39a9cb722e7584256fa5"}
Sep 09 06:56:12.514494 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:12.514053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dq8sm" event={"ID":"9339c265-50ca-4f68-9bef-a6a95a86cb88","Type":"ContainerStarted","Data":"29b2bbceb71524f227511dbcc901b7fe15c6abad40b8983b35805b312b6f1ac7"}
Sep 09 06:56:12.532270 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:12.532223    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dq8sm" podStartSLOduration=1.5322063670000001 podStartE2EDuration="1.532206367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:56:12.530844864 +0000 UTC m=+295.751085976" watchObservedRunningTime="2026-09-09 06:56:12.532206367 +0000 UTC m=+295.752447480"
Sep 09 06:56:17.117939 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:17.117905    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 06:56:17.119934 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:17.119906    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 06:56:17.123742 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:17.123721    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 06:56:19.328756 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.328719    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"]
Sep 09 06:56:19.330687 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.330670    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.333092 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.333067    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 06:56:19.333392 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.333374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 06:56:19.334117 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.334087    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phbjw\""
Sep 09 06:56:19.339950 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.339917    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"]
Sep 09 06:56:19.355996 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.355971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.356095 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.356005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.356095 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.356040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8l76\" (UniqueName: \"kubernetes.io/projected/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-kube-api-access-r8l76\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.456689 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.456656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.456834 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.456699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8l76\" (UniqueName: \"kubernetes.io/projected/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-kube-api-access-r8l76\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.456834 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.456735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.457029 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.457010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.457086 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.457069    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.465301 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.465279    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8l76\" (UniqueName: \"kubernetes.io/projected/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-kube-api-access-r8l76\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.640008 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.639921    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:19.909956 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:19.909873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"]
Sep 09 06:56:19.913735 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:56:19.913693    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cff69eb_dd24_4608_8037_8ddc6f38a3d7.slice/crio-b74a14ae1efcaa5919704eefbb36650a3fedb28464f3265509d7db3ea7daa43e WatchSource:0}: Error finding container b74a14ae1efcaa5919704eefbb36650a3fedb28464f3265509d7db3ea7daa43e: Status 404 returned error can't find the container with id b74a14ae1efcaa5919704eefbb36650a3fedb28464f3265509d7db3ea7daa43e
Sep 09 06:56:20.547431 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:20.547392    2562 generic.go:356] "Generic (PLEG): container finished" podID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerID="0b57e3c914f6927f26b937740f2b9bbe7b2412fd1793ccf2fcbf922311dda4e4" exitCode=0
Sep 09 06:56:20.547800 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:20.547465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm" event={"ID":"2cff69eb-dd24-4608-8037-8ddc6f38a3d7","Type":"ContainerDied","Data":"0b57e3c914f6927f26b937740f2b9bbe7b2412fd1793ccf2fcbf922311dda4e4"}
Sep 09 06:56:20.547800 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:20.547517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm" event={"ID":"2cff69eb-dd24-4608-8037-8ddc6f38a3d7","Type":"ContainerStarted","Data":"b74a14ae1efcaa5919704eefbb36650a3fedb28464f3265509d7db3ea7daa43e"}
Sep 09 06:56:20.547964 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:20.547950    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 06:56:21.553116 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:21.553080    2562 generic.go:356] "Generic (PLEG): container finished" podID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerID="0fe70183579beb1c823f8dfecbf1baf011183e12bbfe21cd6b1157ee33b62713" exitCode=0
Sep 09 06:56:21.553504 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:21.553146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm" event={"ID":"2cff69eb-dd24-4608-8037-8ddc6f38a3d7","Type":"ContainerDied","Data":"0fe70183579beb1c823f8dfecbf1baf011183e12bbfe21cd6b1157ee33b62713"}
Sep 09 06:56:22.559147 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:22.559092    2562 generic.go:356] "Generic (PLEG): container finished" podID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerID="95a03cfd9d788ea448ce724f465ac4344db00153eededd11ccd15a69117985a6" exitCode=0
Sep 09 06:56:22.559588 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:22.559175    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm" event={"ID":"2cff69eb-dd24-4608-8037-8ddc6f38a3d7","Type":"ContainerDied","Data":"95a03cfd9d788ea448ce724f465ac4344db00153eededd11ccd15a69117985a6"}
Sep 09 06:56:23.683873 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.683852    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:23.787225 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.787187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-util\") pod \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") "
Sep 09 06:56:23.787225 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.787230    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-bundle\") pod \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") "
Sep 09 06:56:23.787488 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.787300    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8l76\" (UniqueName: \"kubernetes.io/projected/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-kube-api-access-r8l76\") pod \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\" (UID: \"2cff69eb-dd24-4608-8037-8ddc6f38a3d7\") "
Sep 09 06:56:23.787948 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.787913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-bundle" (OuterVolumeSpecName: "bundle") pod "2cff69eb-dd24-4608-8037-8ddc6f38a3d7" (UID: "2cff69eb-dd24-4608-8037-8ddc6f38a3d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:56:23.789392 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.789365    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-kube-api-access-r8l76" (OuterVolumeSpecName: "kube-api-access-r8l76") pod "2cff69eb-dd24-4608-8037-8ddc6f38a3d7" (UID: "2cff69eb-dd24-4608-8037-8ddc6f38a3d7"). InnerVolumeSpecName "kube-api-access-r8l76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:56:23.793051 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.793028    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-util" (OuterVolumeSpecName: "util") pod "2cff69eb-dd24-4608-8037-8ddc6f38a3d7" (UID: "2cff69eb-dd24-4608-8037-8ddc6f38a3d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:56:23.887929 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.887841    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8l76\" (UniqueName: \"kubernetes.io/projected/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-kube-api-access-r8l76\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:23.887929 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.887873    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-util\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:23.887929 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:23.887885    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cff69eb-dd24-4608-8037-8ddc6f38a3d7-bundle\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:24.568620 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:24.568592    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm"
Sep 09 06:56:24.568803 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:24.568584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhcbxm" event={"ID":"2cff69eb-dd24-4608-8037-8ddc6f38a3d7","Type":"ContainerDied","Data":"b74a14ae1efcaa5919704eefbb36650a3fedb28464f3265509d7db3ea7daa43e"}
Sep 09 06:56:24.568803 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:24.568716    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b74a14ae1efcaa5919704eefbb36650a3fedb28464f3265509d7db3ea7daa43e"
Sep 09 06:56:33.740906 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.740865    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"]
Sep 09 06:56:33.741428 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.741255    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerName="util"
Sep 09 06:56:33.741428 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.741273    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerName="util"
Sep 09 06:56:33.741428 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.741303    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerName="pull"
Sep 09 06:56:33.741428 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.741312    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerName="pull"
Sep 09 06:56:33.741428 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.741324    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerName="extract"
Sep 09 06:56:33.741428 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.741333    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerName="extract"
Sep 09 06:56:33.741428 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.741398    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cff69eb-dd24-4608-8037-8ddc6f38a3d7" containerName="extract"
Sep 09 06:56:33.784351 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.784314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"]
Sep 09 06:56:33.784522 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.784426    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.794990 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.794886    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 06:56:33.795557 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.795535    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 06:56:33.795679 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.795557    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 06:56:33.795679 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.795586    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 06:56:33.795800 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.795772    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 06:56:33.795852 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.795820    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9blcl\""
Sep 09 06:56:33.863338 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.863303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54d659b4-aea7-49ef-9633-7942680d04fb-manager-config\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.863338 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.863338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54d659b4-aea7-49ef-9633-7942680d04fb-cert\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.863525 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.863383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54d659b4-aea7-49ef-9633-7942680d04fb-metrics-cert\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.863525 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.863401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgz86\" (UniqueName: \"kubernetes.io/projected/54d659b4-aea7-49ef-9633-7942680d04fb-kube-api-access-pgz86\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.964425 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.964387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54d659b4-aea7-49ef-9633-7942680d04fb-metrics-cert\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.964425 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.964424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgz86\" (UniqueName: \"kubernetes.io/projected/54d659b4-aea7-49ef-9633-7942680d04fb-kube-api-access-pgz86\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.964668 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.964452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54d659b4-aea7-49ef-9633-7942680d04fb-manager-config\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.964668 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.964475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54d659b4-aea7-49ef-9633-7942680d04fb-cert\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.965196 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.965171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54d659b4-aea7-49ef-9633-7942680d04fb-manager-config\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.966865 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.966843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54d659b4-aea7-49ef-9633-7942680d04fb-cert\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.966948 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.966898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/54d659b4-aea7-49ef-9633-7942680d04fb-metrics-cert\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:33.974757 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:33.974733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgz86\" (UniqueName: \"kubernetes.io/projected/54d659b4-aea7-49ef-9633-7942680d04fb-kube-api-access-pgz86\") pod \"lws-controller-manager-789c5bdf9c-q4jj2\" (UID: \"54d659b4-aea7-49ef-9633-7942680d04fb\") " pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:34.097256 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:34.097174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:34.389249 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:34.389175    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"]
Sep 09 06:56:34.393304 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:56:34.393277    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d659b4_aea7_49ef_9633_7942680d04fb.slice/crio-34bffbf368ad5b0adc5454ee9cfe5a93e2b7f00dd9b268b0f1344836898bf1bd WatchSource:0}: Error finding container 34bffbf368ad5b0adc5454ee9cfe5a93e2b7f00dd9b268b0f1344836898bf1bd: Status 404 returned error can't find the container with id 34bffbf368ad5b0adc5454ee9cfe5a93e2b7f00dd9b268b0f1344836898bf1bd
Sep 09 06:56:34.611440 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:34.611403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2" event={"ID":"54d659b4-aea7-49ef-9633-7942680d04fb","Type":"ContainerStarted","Data":"34bffbf368ad5b0adc5454ee9cfe5a93e2b7f00dd9b268b0f1344836898bf1bd"}
Sep 09 06:56:35.779278 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.779247    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"]
Sep 09 06:56:35.781450 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.781433    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.784050 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.784023    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 06:56:35.784195 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.784150    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phbjw\""
Sep 09 06:56:35.784982 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.784963    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 06:56:35.796083 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.796051    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"]
Sep 09 06:56:35.876358 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.876330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.876475 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.876369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.876475 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.876400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpll\" (UniqueName: \"kubernetes.io/projected/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-kube-api-access-kdpll\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.976696 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.976663    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpll\" (UniqueName: \"kubernetes.io/projected/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-kube-api-access-kdpll\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.976848 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.976707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.976848 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.976832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.977070 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.977049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.977154 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.977141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:35.996405 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:35.996381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpll\" (UniqueName: \"kubernetes.io/projected/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-kube-api-access-kdpll\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:36.090800 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.090726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:36.362246 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.362170    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"]
Sep 09 06:56:36.365030 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:56:36.364999    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd4ffc7_e97c_4113_8f9a_84d98c7dc2cb.slice/crio-40457458cca1c0595f5ffad604d9f5b33b677c70599a789a20839a7cb163a33e WatchSource:0}: Error finding container 40457458cca1c0595f5ffad604d9f5b33b677c70599a789a20839a7cb163a33e: Status 404 returned error can't find the container with id 40457458cca1c0595f5ffad604d9f5b33b677c70599a789a20839a7cb163a33e
Sep 09 06:56:36.622186 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.622078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2" event={"ID":"54d659b4-aea7-49ef-9633-7942680d04fb","Type":"ContainerStarted","Data":"c89c0ec87121e58e73bba20982390376b3a3aab4c47d474e9c2a0ecd9651d185"}
Sep 09 06:56:36.622351 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.622247    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:36.623618 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.623590    2562 generic.go:356] "Generic (PLEG): container finished" podID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerID="3cba05343c6b897eb8bafd370f72e4538acfd1e204282db1bccf37028e3d5b6b" exitCode=0
Sep 09 06:56:36.623750 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.623633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn" event={"ID":"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb","Type":"ContainerDied","Data":"3cba05343c6b897eb8bafd370f72e4538acfd1e204282db1bccf37028e3d5b6b"}
Sep 09 06:56:36.623750 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.623670    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn" event={"ID":"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb","Type":"ContainerStarted","Data":"40457458cca1c0595f5ffad604d9f5b33b677c70599a789a20839a7cb163a33e"}
Sep 09 06:56:36.653397 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.653343    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2" podStartSLOduration=2.241247087 podStartE2EDuration="3.65332857s" totalImagesPullingTime="1.412081483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:56:36.653090891 +0000 UTC m=+319.873332005" watchObservedRunningTime="2026-09-09 06:56:36.65332857 +0000 UTC m=+319.873569683"
Sep 09 06:56:36.745060 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.745014    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"]
Sep 09 06:56:36.747064 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.747048    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.755422 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.755401    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 06:56:36.755749 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.755731    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 06:56:36.755845 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.755788    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q96d5\""
Sep 09 06:56:36.755845 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.755735    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 06:56:36.755845 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.755739    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 06:56:36.777259 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.777235    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"]
Sep 09 06:56:36.782727 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.782704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzq2k\" (UniqueName: \"kubernetes.io/projected/8ec7696c-49a2-4199-a2e0-e14c1611923a-kube-api-access-gzq2k\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.783037 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.782754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ec7696c-49a2-4199-a2e0-e14c1611923a-tmp\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.783037 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.782790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ec7696c-49a2-4199-a2e0-e14c1611923a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.783037 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.782817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ec7696c-49a2-4199-a2e0-e14c1611923a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.783037 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.782854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ec7696c-49a2-4199-a2e0-e14c1611923a-manifests\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.884152 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.884049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ec7696c-49a2-4199-a2e0-e14c1611923a-tmp\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.884152 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.884088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ec7696c-49a2-4199-a2e0-e14c1611923a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.884152 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.884135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ec7696c-49a2-4199-a2e0-e14c1611923a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.884420 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.884171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ec7696c-49a2-4199-a2e0-e14c1611923a-manifests\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.884420 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.884215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzq2k\" (UniqueName: \"kubernetes.io/projected/8ec7696c-49a2-4199-a2e0-e14c1611923a-kube-api-access-gzq2k\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.884587 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.884561    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ec7696c-49a2-4199-a2e0-e14c1611923a-manifests\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.886430 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.886406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ec7696c-49a2-4199-a2e0-e14c1611923a-tmp\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.886535 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.886515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ec7696c-49a2-4199-a2e0-e14c1611923a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.886587 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.886535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ec7696c-49a2-4199-a2e0-e14c1611923a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:36.902961 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:36.902935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzq2k\" (UniqueName: \"kubernetes.io/projected/8ec7696c-49a2-4199-a2e0-e14c1611923a-kube-api-access-gzq2k\") pod \"opendatahub-operator-controller-manager-7c69cbcb-xbxsv\" (UID: \"8ec7696c-49a2-4199-a2e0-e14c1611923a\") " pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:37.056422 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:37.056384    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:37.339413 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:37.339389    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"]
Sep 09 06:56:37.343350 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:56:37.343330    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec7696c_49a2_4199_a2e0_e14c1611923a.slice/crio-a8d3ad5df34c5d0a5fbf65880418f39dc178885d85d20ba519e8d77b23d70685 WatchSource:0}: Error finding container a8d3ad5df34c5d0a5fbf65880418f39dc178885d85d20ba519e8d77b23d70685: Status 404 returned error can't find the container with id a8d3ad5df34c5d0a5fbf65880418f39dc178885d85d20ba519e8d77b23d70685
Sep 09 06:56:37.629506 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:37.629410    2562 generic.go:356] "Generic (PLEG): container finished" podID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerID="8c810d3f3992971a5373e1457679e699cbc0432a4e46ea443b5356a19a0fe538" exitCode=0
Sep 09 06:56:37.629506 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:37.629442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn" event={"ID":"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb","Type":"ContainerDied","Data":"8c810d3f3992971a5373e1457679e699cbc0432a4e46ea443b5356a19a0fe538"}
Sep 09 06:56:37.631115 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:37.630946    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv" event={"ID":"8ec7696c-49a2-4199-a2e0-e14c1611923a","Type":"ContainerStarted","Data":"a8d3ad5df34c5d0a5fbf65880418f39dc178885d85d20ba519e8d77b23d70685"}
Sep 09 06:56:37.782739 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:56:37.782709    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd4ffc7_e97c_4113_8f9a_84d98c7dc2cb.slice/crio-68406f4cffa0c4c7fd8679bbbe2cd76eda680a830c94815147e53cd0a00571eb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd4ffc7_e97c_4113_8f9a_84d98c7dc2cb.slice/crio-conmon-68406f4cffa0c4c7fd8679bbbe2cd76eda680a830c94815147e53cd0a00571eb.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 06:56:38.639744 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:38.639684    2562 generic.go:356] "Generic (PLEG): container finished" podID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerID="68406f4cffa0c4c7fd8679bbbe2cd76eda680a830c94815147e53cd0a00571eb" exitCode=0
Sep 09 06:56:38.639907 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:38.639800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn" event={"ID":"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb","Type":"ContainerDied","Data":"68406f4cffa0c4c7fd8679bbbe2cd76eda680a830c94815147e53cd0a00571eb"}
Sep 09 06:56:40.368972 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.368941    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:40.415248 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.415223    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-util\") pod \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") "
Sep 09 06:56:40.415356 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.415285    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-bundle\") pod \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") "
Sep 09 06:56:40.415356 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.415304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdpll\" (UniqueName: \"kubernetes.io/projected/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-kube-api-access-kdpll\") pod \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\" (UID: \"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb\") "
Sep 09 06:56:40.415935 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.415911    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-bundle" (OuterVolumeSpecName: "bundle") pod "4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" (UID: "4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:56:40.417268 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.417249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-kube-api-access-kdpll" (OuterVolumeSpecName: "kube-api-access-kdpll") pod "4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" (UID: "4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb"). InnerVolumeSpecName "kube-api-access-kdpll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:56:40.420268 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.420236    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-util" (OuterVolumeSpecName: "util") pod "4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" (UID: "4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:56:40.516666 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.516640    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-bundle\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:40.516666 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.516664    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpll\" (UniqueName: \"kubernetes.io/projected/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-kube-api-access-kdpll\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:40.516838 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.516674    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb-util\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:56:40.650587 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.650493    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn"
Sep 09 06:56:40.650587 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.650508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tckcn" event={"ID":"4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb","Type":"ContainerDied","Data":"40457458cca1c0595f5ffad604d9f5b33b677c70599a789a20839a7cb163a33e"}
Sep 09 06:56:40.650587 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.650562    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40457458cca1c0595f5ffad604d9f5b33b677c70599a789a20839a7cb163a33e"
Sep 09 06:56:40.652160 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:40.652132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv" event={"ID":"8ec7696c-49a2-4199-a2e0-e14c1611923a","Type":"ContainerStarted","Data":"63ff697a685dafebb636a1e3911338484646d79533020f0fbdf25b27ba7896cd"}
Sep 09 06:56:41.658262 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:41.658225    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ec7696c-49a2-4199-a2e0-e14c1611923a" containerID="63ff697a685dafebb636a1e3911338484646d79533020f0fbdf25b27ba7896cd" exitCode=0
Sep 09 06:56:41.658610 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:41.658309    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv" event={"ID":"8ec7696c-49a2-4199-a2e0-e14c1611923a","Type":"ContainerDied","Data":"63ff697a685dafebb636a1e3911338484646d79533020f0fbdf25b27ba7896cd"}
Sep 09 06:56:42.664752 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:42.664719    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv" event={"ID":"8ec7696c-49a2-4199-a2e0-e14c1611923a","Type":"ContainerStarted","Data":"94c9efcab7528b6e81ecb44cd0f9818bd120808d54a04ba621404e812fd7e05b"}
Sep 09 06:56:42.665244 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:42.664846    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:56:42.694727 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:42.694649    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv" podStartSLOduration=2.205923374 podStartE2EDuration="6.694632485s" totalImagesPullingTime="3.468356218s" totalInitContainerRuntime="1.020352893s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:56:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:56:42.692562752 +0000 UTC m=+325.912803887" watchObservedRunningTime="2026-09-09 06:56:42.694632485 +0000 UTC m=+325.914873598"
Sep 09 06:56:47.633952 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:47.633915    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-789c5bdf9c-q4jj2"
Sep 09 06:56:53.671029 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:56:53.670996    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c69cbcb-xbxsv"
Sep 09 06:57:05.072438 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.072401    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"]
Sep 09 06:57:05.072903 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.072694    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerName="extract"
Sep 09 06:57:05.072903 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.072706    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerName="extract"
Sep 09 06:57:05.072903 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.072720    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerName="util"
Sep 09 06:57:05.072903 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.072725    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerName="util"
Sep 09 06:57:05.072903 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.072735    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerName="pull"
Sep 09 06:57:05.072903 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.072744    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerName="pull"
Sep 09 06:57:05.072903 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.072799    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd4ffc7-e97c-4113-8f9a-84d98c7dc2cb" containerName="extract"
Sep 09 06:57:05.080858 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.080832    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.083734 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.083706    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 06:57:05.083878 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.083833    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x9zw4\""
Sep 09 06:57:05.083950 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.083925    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 06:57:05.084037 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.084021    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 06:57:05.089565 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.089539    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"]
Sep 09 06:57:05.211295 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.211295 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07207665-cc66-4471-819a-c1f6e666e3d5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.211662 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.211662 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07207665-cc66-4471-819a-c1f6e666e3d5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.211662 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.211662 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07207665-cc66-4471-819a-c1f6e666e3d5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.211662 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.211662 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9d5\" (UniqueName: \"kubernetes.io/projected/07207665-cc66-4471-819a-c1f6e666e3d5-kube-api-access-bm9d5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.211880 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.211667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312418 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312418 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07207665-cc66-4471-819a-c1f6e666e3d5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07207665-cc66-4471-819a-c1f6e666e3d5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07207665-cc66-4471-819a-c1f6e666e3d5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9d5\" (UniqueName: \"kubernetes.io/projected/07207665-cc66-4471-819a-c1f6e666e3d5-kube-api-access-bm9d5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312984 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312858    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312984 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.312984 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.312936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.313077 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.313056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.313465 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.313450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07207665-cc66-4471-819a-c1f6e666e3d5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.314865 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.314849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07207665-cc66-4471-819a-c1f6e666e3d5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.314980 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.314964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07207665-cc66-4471-819a-c1f6e666e3d5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.320955 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.320933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07207665-cc66-4471-819a-c1f6e666e3d5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.321286 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.321262    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9d5\" (UniqueName: \"kubernetes.io/projected/07207665-cc66-4471-819a-c1f6e666e3d5-kube-api-access-bm9d5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw\" (UID: \"07207665-cc66-4471-819a-c1f6e666e3d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.393119 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.393003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:05.670840 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.670811    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"]
Sep 09 06:57:05.672561 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:57:05.672537    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07207665_cc66_4471_819a_c1f6e666e3d5.slice/crio-7fd89f755aaaf79f55d50c8ecd2300ede7bcf8ae4fcc3fe03183130f71b55431 WatchSource:0}: Error finding container 7fd89f755aaaf79f55d50c8ecd2300ede7bcf8ae4fcc3fe03183130f71b55431: Status 404 returned error can't find the container with id 7fd89f755aaaf79f55d50c8ecd2300ede7bcf8ae4fcc3fe03183130f71b55431
Sep 09 06:57:05.763713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:05.763678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw" event={"ID":"07207665-cc66-4471-819a-c1f6e666e3d5","Type":"ContainerStarted","Data":"7fd89f755aaaf79f55d50c8ecd2300ede7bcf8ae4fcc3fe03183130f71b55431"}
Sep 09 06:57:08.088924 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.088877    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ggm6p"]
Sep 09 06:57:08.098538 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.098512    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:08.101997 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.101968    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 06:57:08.101997 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.101982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dhnd9\""
Sep 09 06:57:08.109315 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.108449    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ggm6p"]
Sep 09 06:57:08.236221 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.236182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl55d\" (UniqueName: \"kubernetes.io/projected/c4aca7f0-6081-47d3-8195-918da75f5209-kube-api-access-zl55d\") pod \"odh-model-controller-7998c67dc4-ggm6p\" (UID: \"c4aca7f0-6081-47d3-8195-918da75f5209\") " pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:08.236379 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.236251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4aca7f0-6081-47d3-8195-918da75f5209-cert\") pod \"odh-model-controller-7998c67dc4-ggm6p\" (UID: \"c4aca7f0-6081-47d3-8195-918da75f5209\") " pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:08.309015 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.308967    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 06:57:08.309172 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.309076    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 06:57:08.309172 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.309141    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 06:57:08.336741 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.336719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl55d\" (UniqueName: \"kubernetes.io/projected/c4aca7f0-6081-47d3-8195-918da75f5209-kube-api-access-zl55d\") pod \"odh-model-controller-7998c67dc4-ggm6p\" (UID: \"c4aca7f0-6081-47d3-8195-918da75f5209\") " pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:08.336837 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.336774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4aca7f0-6081-47d3-8195-918da75f5209-cert\") pod \"odh-model-controller-7998c67dc4-ggm6p\" (UID: \"c4aca7f0-6081-47d3-8195-918da75f5209\") " pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:08.336960 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:57:08.336938    2562 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 06:57:08.337038 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:57:08.337024    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4aca7f0-6081-47d3-8195-918da75f5209-cert podName:c4aca7f0-6081-47d3-8195-918da75f5209 nodeName:}" failed. No retries permitted until 2026-09-09 06:57:08.837003302 +0000 UTC m=+352.057244408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c4aca7f0-6081-47d3-8195-918da75f5209-cert") pod "odh-model-controller-7998c67dc4-ggm6p" (UID: "c4aca7f0-6081-47d3-8195-918da75f5209") : secret "odh-model-controller-webhook-cert" not found
Sep 09 06:57:08.351398 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.351377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl55d\" (UniqueName: \"kubernetes.io/projected/c4aca7f0-6081-47d3-8195-918da75f5209-kube-api-access-zl55d\") pod \"odh-model-controller-7998c67dc4-ggm6p\" (UID: \"c4aca7f0-6081-47d3-8195-918da75f5209\") " pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:08.782735 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.782693    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw" event={"ID":"07207665-cc66-4471-819a-c1f6e666e3d5","Type":"ContainerStarted","Data":"54b3829d9dac64247bab79bb1e14bd750f134644633697b947bce908703d9593"}
Sep 09 06:57:08.819170 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.819092    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw" podStartSLOduration=1.18497682 podStartE2EDuration="3.819074265s" totalImagesPullingTime="2.634097445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:57:08.817849643 +0000 UTC m=+352.038090752" watchObservedRunningTime="2026-09-09 06:57:08.819074265 +0000 UTC m=+352.039315379"
Sep 09 06:57:08.840978 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.840947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4aca7f0-6081-47d3-8195-918da75f5209-cert\") pod \"odh-model-controller-7998c67dc4-ggm6p\" (UID: \"c4aca7f0-6081-47d3-8195-918da75f5209\") " pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:08.843450 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:08.843423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4aca7f0-6081-47d3-8195-918da75f5209-cert\") pod \"odh-model-controller-7998c67dc4-ggm6p\" (UID: \"c4aca7f0-6081-47d3-8195-918da75f5209\") " pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:09.015346 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:09.015314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:09.292721 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:09.292551    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ggm6p"]
Sep 09 06:57:09.295357 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:57:09.295319    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4aca7f0_6081_47d3_8195_918da75f5209.slice/crio-cb748e4464f19b1f10ee3331734f781f96595c68aa268c147358cdc9c1a12314 WatchSource:0}: Error finding container cb748e4464f19b1f10ee3331734f781f96595c68aa268c147358cdc9c1a12314: Status 404 returned error can't find the container with id cb748e4464f19b1f10ee3331734f781f96595c68aa268c147358cdc9c1a12314
Sep 09 06:57:09.393477 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:09.393390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:09.412576 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:09.412546    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:09.789411 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:09.789366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p" event={"ID":"c4aca7f0-6081-47d3-8195-918da75f5209","Type":"ContainerStarted","Data":"cb748e4464f19b1f10ee3331734f781f96595c68aa268c147358cdc9c1a12314"}
Sep 09 06:57:09.789605 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:09.789591    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:09.790684 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:09.790662    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw"
Sep 09 06:57:12.807695 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:12.807600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p" event={"ID":"c4aca7f0-6081-47d3-8195-918da75f5209","Type":"ContainerStarted","Data":"1cd962080cdbaf992fcdcd83f9bf189b969c54a84b0b5acc4b592fff23aeee0a"}
Sep 09 06:57:12.808119 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:12.807713    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:12.827522 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:12.826926    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p" podStartSLOduration=1.687942397 podStartE2EDuration="4.82690642s" totalImagesPullingTime="3.138964023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:57:12.8241643 +0000 UTC m=+356.044405414" watchObservedRunningTime="2026-09-09 06:57:12.82690642 +0000 UTC m=+356.047147534"
Sep 09 06:57:18.279299 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.279261    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"]
Sep 09 06:57:18.283026 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.283003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:18.286332 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.286306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 06:57:18.287142 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.287042    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-g8bf9\""
Sep 09 06:57:18.312283 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.312253    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"]
Sep 09 06:57:18.313896 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.313869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-72c9j\" (UID: \"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:18.314179 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.314155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbzs\" (UniqueName: \"kubernetes.io/projected/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-kube-api-access-mgbzs\") pod \"llmisvc-controller-manager-777c6bd7d5-72c9j\" (UID: \"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:18.415214 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.415177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-72c9j\" (UID: \"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:18.415401 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.415274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbzs\" (UniqueName: \"kubernetes.io/projected/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-kube-api-access-mgbzs\") pod \"llmisvc-controller-manager-777c6bd7d5-72c9j\" (UID: \"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:18.415401 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:57:18.415335    2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 09 06:57:18.415401 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:57:18.415395    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-cert podName:b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef nodeName:}" failed. No retries permitted until 2026-09-09 06:57:18.915379623 +0000 UTC m=+362.135620719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-cert") pod "llmisvc-controller-manager-777c6bd7d5-72c9j" (UID: "b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef") : secret "llmisvc-webhook-server-cert" not found
Sep 09 06:57:18.425051 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.425023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbzs\" (UniqueName: \"kubernetes.io/projected/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-kube-api-access-mgbzs\") pod \"llmisvc-controller-manager-777c6bd7d5-72c9j\" (UID: \"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:18.920001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.919968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-72c9j\" (UID: \"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:18.922519 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:18.922492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-72c9j\" (UID: \"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:19.197831 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:19.197732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:19.468430 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:19.468405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"]
Sep 09 06:57:19.469868 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:57:19.469839    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb3ef9da1_c7d6_4e0a_b877_78fdbaef83ef.slice/crio-a745f94e70926d130795699f7d76e4e262ab77dc4d5dcb5b8d97828a999071b3 WatchSource:0}: Error finding container a745f94e70926d130795699f7d76e4e262ab77dc4d5dcb5b8d97828a999071b3: Status 404 returned error can't find the container with id a745f94e70926d130795699f7d76e4e262ab77dc4d5dcb5b8d97828a999071b3
Sep 09 06:57:19.838929 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:19.838836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j" event={"ID":"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef","Type":"ContainerStarted","Data":"a745f94e70926d130795699f7d76e4e262ab77dc4d5dcb5b8d97828a999071b3"}
Sep 09 06:57:22.855654 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:22.855614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j" event={"ID":"b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef","Type":"ContainerStarted","Data":"d0738523e7bd36f575189585a6b7fc49c2de910a0e074314b8b849389d352ce4"}
Sep 09 06:57:22.856032 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:22.855735    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:57:22.887337 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:22.887285    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j" podStartSLOduration=2.170942307 podStartE2EDuration="4.88726895s" totalImagesPullingTime="2.716326643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:57:22.885146803 +0000 UTC m=+366.105387915" watchObservedRunningTime="2026-09-09 06:57:22.88726895 +0000 UTC m=+366.107510063"
Sep 09 06:57:23.863655 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:23.863621    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ggm6p"
Sep 09 06:57:53.864281 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:57:53.864246    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-72c9j"
Sep 09 06:58:13.209194 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.209154    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"]
Sep 09 06:58:13.211491 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.211474    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.214156 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.214129    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 06:58:13.215009 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.214992    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 06:58:13.215136 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.215011    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-phbjw\""
Sep 09 06:58:13.220809 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.220788    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"]
Sep 09 06:58:13.305327 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.305296    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"]
Sep 09 06:58:13.307645 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.307627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.318921 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.318892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"]
Sep 09 06:58:13.358145 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.358098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j78d\" (UniqueName: \"kubernetes.io/projected/428a680f-e622-4cae-bbca-d1cc58b6d1b9-kube-api-access-7j78d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.358291 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.358161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.358291 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.358235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.416741 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.416712    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"]
Sep 09 06:58:13.419015 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.418997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.434083 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.434053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"]
Sep 09 06:58:13.459692 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.459611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.459692 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.459686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.459881 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.459735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m258h\" (UniqueName: \"kubernetes.io/projected/eebdc326-181f-4562-8a32-4b054c249218-kube-api-access-m258h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.459881 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.459813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.459881 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.459849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.460020 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.459881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j78d\" (UniqueName: \"kubernetes.io/projected/428a680f-e622-4cae-bbca-d1cc58b6d1b9-kube-api-access-7j78d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.463713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.460596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.463713 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.460638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.469457 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.469435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j78d\" (UniqueName: \"kubernetes.io/projected/428a680f-e622-4cae-bbca-d1cc58b6d1b9-kube-api-access-7j78d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.508070 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.508035    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"]
Sep 09 06:58:13.510670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.510654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.521303 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.521175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:13.522090 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.522060    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"]
Sep 09 06:58:13.561277 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.561245    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.561429 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.561288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m258h\" (UniqueName: \"kubernetes.io/projected/eebdc326-181f-4562-8a32-4b054c249218-kube-api-access-m258h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.561429 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.561314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.561429 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.561350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.561429 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.561375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.561620 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.561424    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kgrb\" (UniqueName: \"kubernetes.io/projected/3296c90b-f0d9-4761-9719-de5ada54109f-kube-api-access-6kgrb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.561693 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.561678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.561803 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.561786    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.571990 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.571956    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m258h\" (UniqueName: \"kubernetes.io/projected/eebdc326-181f-4562-8a32-4b054c249218-kube-api-access-m258h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.616648 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.616616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:13.662562 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.662521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.662714 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.662594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.662714 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.662644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.662714 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.662681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9skc\" (UniqueName: \"kubernetes.io/projected/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-kube-api-access-c9skc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.662818 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.662737    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.662818 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.662767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgrb\" (UniqueName: \"kubernetes.io/projected/3296c90b-f0d9-4761-9719-de5ada54109f-kube-api-access-6kgrb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.663085 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.663060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.663155 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.663087    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.679120 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.679072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kgrb\" (UniqueName: \"kubernetes.io/projected/3296c90b-f0d9-4761-9719-de5ada54109f-kube-api-access-6kgrb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.727529 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.727453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:13.764318 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.764282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.764481 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.764353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.764481 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.764427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9skc\" (UniqueName: \"kubernetes.io/projected/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-kube-api-access-c9skc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.764815 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.764784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.765093 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.765055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.777600 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.777545    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9skc\" (UniqueName: \"kubernetes.io/projected/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-kube-api-access-c9skc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.798801 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.798739    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"]
Sep 09 06:58:13.800250 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:58:13.800223    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428a680f_e622_4cae_bbca_d1cc58b6d1b9.slice/crio-303d228d77f359063f914cd2c838f6f03d5fbfd7d0fefce8b9b160f6f6af72a6 WatchSource:0}: Error finding container 303d228d77f359063f914cd2c838f6f03d5fbfd7d0fefce8b9b160f6f6af72a6: Status 404 returned error can't find the container with id 303d228d77f359063f914cd2c838f6f03d5fbfd7d0fefce8b9b160f6f6af72a6
Sep 09 06:58:13.820742 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.820721    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:13.906721 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:13.906697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"]
Sep 09 06:58:13.908711 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:58:13.908684    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebdc326_181f_4562_8a32_4b054c249218.slice/crio-4a0402e10d198441c6065b52ccf505d15189023b5b14f10cb74868070b450f10 WatchSource:0}: Error finding container 4a0402e10d198441c6065b52ccf505d15189023b5b14f10cb74868070b450f10: Status 404 returned error can't find the container with id 4a0402e10d198441c6065b52ccf505d15189023b5b14f10cb74868070b450f10
Sep 09 06:58:14.012515 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.012493    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"]
Sep 09 06:58:14.015037 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:58:14.015009    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3296c90b_f0d9_4761_9719_de5ada54109f.slice/crio-49a457de9e0ada80e648fb5819162bfbc81a832c0a0db9bdf45663c9a2762b6e WatchSource:0}: Error finding container 49a457de9e0ada80e648fb5819162bfbc81a832c0a0db9bdf45663c9a2762b6e: Status 404 returned error can't find the container with id 49a457de9e0ada80e648fb5819162bfbc81a832c0a0db9bdf45663c9a2762b6e
Sep 09 06:58:14.095132 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.095080    2562 generic.go:356] "Generic (PLEG): container finished" podID="eebdc326-181f-4562-8a32-4b054c249218" containerID="0c4ef2dac05d652518237d58f95c6211d318df36d9e54f92938ada4e605b1629" exitCode=0
Sep 09 06:58:14.095271 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.095133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc" event={"ID":"eebdc326-181f-4562-8a32-4b054c249218","Type":"ContainerDied","Data":"0c4ef2dac05d652518237d58f95c6211d318df36d9e54f92938ada4e605b1629"}
Sep 09 06:58:14.095271 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.095185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc" event={"ID":"eebdc326-181f-4562-8a32-4b054c249218","Type":"ContainerStarted","Data":"4a0402e10d198441c6065b52ccf505d15189023b5b14f10cb74868070b450f10"}
Sep 09 06:58:14.097167 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.097138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj" event={"ID":"3296c90b-f0d9-4761-9719-de5ada54109f","Type":"ContainerStarted","Data":"dc770943331ee5102db5611be25c7a63bab116f9f16acd949767eb6229438dc6"}
Sep 09 06:58:14.097299 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.097186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj" event={"ID":"3296c90b-f0d9-4761-9719-de5ada54109f","Type":"ContainerStarted","Data":"49a457de9e0ada80e648fb5819162bfbc81a832c0a0db9bdf45663c9a2762b6e"}
Sep 09 06:58:14.098914 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.098893    2562 generic.go:356] "Generic (PLEG): container finished" podID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerID="3841a39a72f2515891fced620db8a03e8d6d19479b6eba1a5ebb1be104aca837" exitCode=0
Sep 09 06:58:14.099016 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.098940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd" event={"ID":"428a680f-e622-4cae-bbca-d1cc58b6d1b9","Type":"ContainerDied","Data":"3841a39a72f2515891fced620db8a03e8d6d19479b6eba1a5ebb1be104aca837"}
Sep 09 06:58:14.099016 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.098976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd" event={"ID":"428a680f-e622-4cae-bbca-d1cc58b6d1b9","Type":"ContainerStarted","Data":"303d228d77f359063f914cd2c838f6f03d5fbfd7d0fefce8b9b160f6f6af72a6"}
Sep 09 06:58:14.104189 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:14.104168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"]
Sep 09 06:58:14.124970 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:58:14.124944    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfe18d6_03d1_45eb_98eb_4659ed5b6734.slice/crio-931c420fdf9a6e780ca33190b1bd4d1637ff6c8665bbb076f7a465f892b68918 WatchSource:0}: Error finding container 931c420fdf9a6e780ca33190b1bd4d1637ff6c8665bbb076f7a465f892b68918: Status 404 returned error can't find the container with id 931c420fdf9a6e780ca33190b1bd4d1637ff6c8665bbb076f7a465f892b68918
Sep 09 06:58:15.104357 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:15.104326    2562 generic.go:356] "Generic (PLEG): container finished" podID="3296c90b-f0d9-4761-9719-de5ada54109f" containerID="dc770943331ee5102db5611be25c7a63bab116f9f16acd949767eb6229438dc6" exitCode=0
Sep 09 06:58:15.104655 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:15.104406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj" event={"ID":"3296c90b-f0d9-4761-9719-de5ada54109f","Type":"ContainerDied","Data":"dc770943331ee5102db5611be25c7a63bab116f9f16acd949767eb6229438dc6"}
Sep 09 06:58:15.106007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:15.105986    2562 generic.go:356] "Generic (PLEG): container finished" podID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerID="3074651d516fbb442e221508811bce0731546115721d51deab4cc010b2f6d442" exitCode=0
Sep 09 06:58:15.106115 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:15.106019    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq" event={"ID":"3bfe18d6-03d1-45eb-98eb-4659ed5b6734","Type":"ContainerDied","Data":"3074651d516fbb442e221508811bce0731546115721d51deab4cc010b2f6d442"}
Sep 09 06:58:15.106115 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:15.106047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq" event={"ID":"3bfe18d6-03d1-45eb-98eb-4659ed5b6734","Type":"ContainerStarted","Data":"931c420fdf9a6e780ca33190b1bd4d1637ff6c8665bbb076f7a465f892b68918"}
Sep 09 06:58:16.118066 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:16.117972    2562 generic.go:356] "Generic (PLEG): container finished" podID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerID="0ada7c02a3787ae71ec8ac37303d766610afa591579093af9d2f7dae93711e2d" exitCode=0
Sep 09 06:58:16.118531 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:16.118058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd" event={"ID":"428a680f-e622-4cae-bbca-d1cc58b6d1b9","Type":"ContainerDied","Data":"0ada7c02a3787ae71ec8ac37303d766610afa591579093af9d2f7dae93711e2d"}
Sep 09 06:58:16.120119 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:16.120081    2562 generic.go:356] "Generic (PLEG): container finished" podID="eebdc326-181f-4562-8a32-4b054c249218" containerID="bf104d3000d996adb4868cf65b8f713debbd91580add1db66c767d7f8958d867" exitCode=0
Sep 09 06:58:16.120214 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:16.120163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc" event={"ID":"eebdc326-181f-4562-8a32-4b054c249218","Type":"ContainerDied","Data":"bf104d3000d996adb4868cf65b8f713debbd91580add1db66c767d7f8958d867"}
Sep 09 06:58:16.122490 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:16.122466    2562 generic.go:356] "Generic (PLEG): container finished" podID="3296c90b-f0d9-4761-9719-de5ada54109f" containerID="9fbcb15af6b346502c992d99708024ab07004264210e7bf8b0991ab49d0b8f87" exitCode=0
Sep 09 06:58:16.122589 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:16.122509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj" event={"ID":"3296c90b-f0d9-4761-9719-de5ada54109f","Type":"ContainerDied","Data":"9fbcb15af6b346502c992d99708024ab07004264210e7bf8b0991ab49d0b8f87"}
Sep 09 06:58:17.130359 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:17.130282    2562 generic.go:356] "Generic (PLEG): container finished" podID="eebdc326-181f-4562-8a32-4b054c249218" containerID="9295e1884a390654a51c54c4778dc8dfe3734e8122df63c373887c54a88db027" exitCode=0
Sep 09 06:58:17.130735 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:17.130361    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc" event={"ID":"eebdc326-181f-4562-8a32-4b054c249218","Type":"ContainerDied","Data":"9295e1884a390654a51c54c4778dc8dfe3734e8122df63c373887c54a88db027"}
Sep 09 06:58:17.132152 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:17.132131    2562 generic.go:356] "Generic (PLEG): container finished" podID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerID="88ae1b7cf5a890da235746ad26ecdb8765869707035c44cb50f1ffcdbd273c9e" exitCode=0
Sep 09 06:58:17.132257 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:17.132220    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq" event={"ID":"3bfe18d6-03d1-45eb-98eb-4659ed5b6734","Type":"ContainerDied","Data":"88ae1b7cf5a890da235746ad26ecdb8765869707035c44cb50f1ffcdbd273c9e"}
Sep 09 06:58:17.134464 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:17.134436    2562 generic.go:356] "Generic (PLEG): container finished" podID="3296c90b-f0d9-4761-9719-de5ada54109f" containerID="f07120ead816dad88605083a3d93ef23c3530d9d823f124bd6a9d894e1d0edfe" exitCode=0
Sep 09 06:58:17.134603 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:17.134463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj" event={"ID":"3296c90b-f0d9-4761-9719-de5ada54109f","Type":"ContainerDied","Data":"f07120ead816dad88605083a3d93ef23c3530d9d823f124bd6a9d894e1d0edfe"}
Sep 09 06:58:17.136832 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:17.136808    2562 generic.go:356] "Generic (PLEG): container finished" podID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerID="81e9f2fa9ecb3fa3c6085ee80c8a68bcaf758c6d2c6adf2a88238af5be6645f9" exitCode=0
Sep 09 06:58:17.136941 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:17.136841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd" event={"ID":"428a680f-e622-4cae-bbca-d1cc58b6d1b9","Type":"ContainerDied","Data":"81e9f2fa9ecb3fa3c6085ee80c8a68bcaf758c6d2c6adf2a88238af5be6645f9"}
Sep 09 06:58:18.143900 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.143864    2562 generic.go:356] "Generic (PLEG): container finished" podID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerID="f02e34febdf23b2457deeef67e8cd7e80e717b4b3394292b9de261b849992e73" exitCode=0
Sep 09 06:58:18.144317 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.143946    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq" event={"ID":"3bfe18d6-03d1-45eb-98eb-4659ed5b6734","Type":"ContainerDied","Data":"f02e34febdf23b2457deeef67e8cd7e80e717b4b3394292b9de261b849992e73"}
Sep 09 06:58:18.287597 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.287565    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:18.298725 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.298702    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-util\") pod \"eebdc326-181f-4562-8a32-4b054c249218\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") "
Sep 09 06:58:18.298840 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.298786    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-bundle\") pod \"eebdc326-181f-4562-8a32-4b054c249218\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") "
Sep 09 06:58:18.298898 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.298836    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m258h\" (UniqueName: \"kubernetes.io/projected/eebdc326-181f-4562-8a32-4b054c249218-kube-api-access-m258h\") pod \"eebdc326-181f-4562-8a32-4b054c249218\" (UID: \"eebdc326-181f-4562-8a32-4b054c249218\") "
Sep 09 06:58:18.299415 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.299389    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-bundle" (OuterVolumeSpecName: "bundle") pod "eebdc326-181f-4562-8a32-4b054c249218" (UID: "eebdc326-181f-4562-8a32-4b054c249218"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:58:18.302614 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.302587    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebdc326-181f-4562-8a32-4b054c249218-kube-api-access-m258h" (OuterVolumeSpecName: "kube-api-access-m258h") pod "eebdc326-181f-4562-8a32-4b054c249218" (UID: "eebdc326-181f-4562-8a32-4b054c249218"). InnerVolumeSpecName "kube-api-access-m258h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:58:18.308297 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.308121    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-util" (OuterVolumeSpecName: "util") pod "eebdc326-181f-4562-8a32-4b054c249218" (UID: "eebdc326-181f-4562-8a32-4b054c249218"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:58:18.329628 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.329609    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:18.332269 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.332251    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:18.399929 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.399858    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kgrb\" (UniqueName: \"kubernetes.io/projected/3296c90b-f0d9-4761-9719-de5ada54109f-kube-api-access-6kgrb\") pod \"3296c90b-f0d9-4761-9719-de5ada54109f\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") "
Sep 09 06:58:18.399929 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.399897    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j78d\" (UniqueName: \"kubernetes.io/projected/428a680f-e622-4cae-bbca-d1cc58b6d1b9-kube-api-access-7j78d\") pod \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") "
Sep 09 06:58:18.399929 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.399915    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-util\") pod \"3296c90b-f0d9-4761-9719-de5ada54109f\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") "
Sep 09 06:58:18.400242 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.399938    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-bundle\") pod \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") "
Sep 09 06:58:18.400242 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.399968    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-bundle\") pod \"3296c90b-f0d9-4761-9719-de5ada54109f\" (UID: \"3296c90b-f0d9-4761-9719-de5ada54109f\") "
Sep 09 06:58:18.400242 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.400033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-util\") pod \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\" (UID: \"428a680f-e622-4cae-bbca-d1cc58b6d1b9\") "
Sep 09 06:58:18.400398 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.400283    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m258h\" (UniqueName: \"kubernetes.io/projected/eebdc326-181f-4562-8a32-4b054c249218-kube-api-access-m258h\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:18.400398 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.400302    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-util\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:18.400398 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.400339    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eebdc326-181f-4562-8a32-4b054c249218-bundle\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:18.400585 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.400560    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-bundle" (OuterVolumeSpecName: "bundle") pod "3296c90b-f0d9-4761-9719-de5ada54109f" (UID: "3296c90b-f0d9-4761-9719-de5ada54109f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:58:18.400788 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.400765    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-bundle" (OuterVolumeSpecName: "bundle") pod "428a680f-e622-4cae-bbca-d1cc58b6d1b9" (UID: "428a680f-e622-4cae-bbca-d1cc58b6d1b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:58:18.402064 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.402042    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3296c90b-f0d9-4761-9719-de5ada54109f-kube-api-access-6kgrb" (OuterVolumeSpecName: "kube-api-access-6kgrb") pod "3296c90b-f0d9-4761-9719-de5ada54109f" (UID: "3296c90b-f0d9-4761-9719-de5ada54109f"). InnerVolumeSpecName "kube-api-access-6kgrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:58:18.402388 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.402366    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428a680f-e622-4cae-bbca-d1cc58b6d1b9-kube-api-access-7j78d" (OuterVolumeSpecName: "kube-api-access-7j78d") pod "428a680f-e622-4cae-bbca-d1cc58b6d1b9" (UID: "428a680f-e622-4cae-bbca-d1cc58b6d1b9"). InnerVolumeSpecName "kube-api-access-7j78d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:58:18.405624 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.405592    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-util" (OuterVolumeSpecName: "util") pod "3296c90b-f0d9-4761-9719-de5ada54109f" (UID: "3296c90b-f0d9-4761-9719-de5ada54109f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:58:18.406181 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.406159    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-util" (OuterVolumeSpecName: "util") pod "428a680f-e622-4cae-bbca-d1cc58b6d1b9" (UID: "428a680f-e622-4cae-bbca-d1cc58b6d1b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:58:18.501473 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.501447    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-bundle\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:18.501473 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.501468    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-util\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:18.501473 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.501477    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kgrb\" (UniqueName: \"kubernetes.io/projected/3296c90b-f0d9-4761-9719-de5ada54109f-kube-api-access-6kgrb\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:18.501654 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.501487    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j78d\" (UniqueName: \"kubernetes.io/projected/428a680f-e622-4cae-bbca-d1cc58b6d1b9-kube-api-access-7j78d\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:18.501654 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.501495    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3296c90b-f0d9-4761-9719-de5ada54109f-util\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:18.501654 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:18.501503    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/428a680f-e622-4cae-bbca-d1cc58b6d1b9-bundle\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:19.150719 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.150690    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd"
Sep 09 06:58:19.150719 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.150695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjfqd" event={"ID":"428a680f-e622-4cae-bbca-d1cc58b6d1b9","Type":"ContainerDied","Data":"303d228d77f359063f914cd2c838f6f03d5fbfd7d0fefce8b9b160f6f6af72a6"}
Sep 09 06:58:19.151195 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.150738    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="303d228d77f359063f914cd2c838f6f03d5fbfd7d0fefce8b9b160f6f6af72a6"
Sep 09 06:58:19.152605 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.152587    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc"
Sep 09 06:58:19.152742 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.152623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2ksgc" event={"ID":"eebdc326-181f-4562-8a32-4b054c249218","Type":"ContainerDied","Data":"4a0402e10d198441c6065b52ccf505d15189023b5b14f10cb74868070b450f10"}
Sep 09 06:58:19.152742 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.152670    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a0402e10d198441c6065b52ccf505d15189023b5b14f10cb74868070b450f10"
Sep 09 06:58:19.154590 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.154572    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj"
Sep 09 06:58:19.154687 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.154585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6qmj" event={"ID":"3296c90b-f0d9-4761-9719-de5ada54109f","Type":"ContainerDied","Data":"49a457de9e0ada80e648fb5819162bfbc81a832c0a0db9bdf45663c9a2762b6e"}
Sep 09 06:58:19.154687 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.154618    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a457de9e0ada80e648fb5819162bfbc81a832c0a0db9bdf45663c9a2762b6e"
Sep 09 06:58:19.282157 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.282132    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:19.308656 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.308633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-bundle\") pod \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") "
Sep 09 06:58:19.308798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.308671    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-util\") pod \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") "
Sep 09 06:58:19.308798 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.308765    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9skc\" (UniqueName: \"kubernetes.io/projected/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-kube-api-access-c9skc\") pod \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\" (UID: \"3bfe18d6-03d1-45eb-98eb-4659ed5b6734\") "
Sep 09 06:58:19.309197 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.309169    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-bundle" (OuterVolumeSpecName: "bundle") pod "3bfe18d6-03d1-45eb-98eb-4659ed5b6734" (UID: "3bfe18d6-03d1-45eb-98eb-4659ed5b6734"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:58:19.310836 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.310809    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-kube-api-access-c9skc" (OuterVolumeSpecName: "kube-api-access-c9skc") pod "3bfe18d6-03d1-45eb-98eb-4659ed5b6734" (UID: "3bfe18d6-03d1-45eb-98eb-4659ed5b6734"). InnerVolumeSpecName "kube-api-access-c9skc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:58:19.409386 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.409314    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9skc\" (UniqueName: \"kubernetes.io/projected/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-kube-api-access-c9skc\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:19.409386 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.409338    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-bundle\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:19.673527 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.673492    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-util" (OuterVolumeSpecName: "util") pod "3bfe18d6-03d1-45eb-98eb-4659ed5b6734" (UID: "3bfe18d6-03d1-45eb-98eb-4659ed5b6734"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 06:58:19.711802 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:19.711775    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bfe18d6-03d1-45eb-98eb-4659ed5b6734-util\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:20.160799 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:20.160714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq" event={"ID":"3bfe18d6-03d1-45eb-98eb-4659ed5b6734","Type":"ContainerDied","Data":"931c420fdf9a6e780ca33190b1bd4d1637ff6c8665bbb076f7a465f892b68918"}
Sep 09 06:58:20.160799 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:20.160738    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lhxnq"
Sep 09 06:58:20.160799 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:20.160770    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931c420fdf9a6e780ca33190b1bd4d1637ff6c8665bbb076f7a465f892b68918"
Sep 09 06:58:27.128458 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128412    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"]
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128702    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerName="util"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128712    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerName="util"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128720    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eebdc326-181f-4562-8a32-4b054c249218" containerName="extract"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128725    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eebdc326-181f-4562-8a32-4b054c249218" containerName="extract"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128735    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3296c90b-f0d9-4761-9719-de5ada54109f" containerName="extract"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128741    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3296c90b-f0d9-4761-9719-de5ada54109f" containerName="extract"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128749    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerName="pull"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128754    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerName="pull"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128760    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3296c90b-f0d9-4761-9719-de5ada54109f" containerName="util"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128765    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3296c90b-f0d9-4761-9719-de5ada54109f" containerName="util"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128771    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerName="pull"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128775    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerName="pull"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128780    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3296c90b-f0d9-4761-9719-de5ada54109f" containerName="pull"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128785    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3296c90b-f0d9-4761-9719-de5ada54109f" containerName="pull"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128793    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eebdc326-181f-4562-8a32-4b054c249218" containerName="pull"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128798    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eebdc326-181f-4562-8a32-4b054c249218" containerName="pull"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128802    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerName="extract"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128807    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerName="extract"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128814    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerName="util"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128818    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerName="util"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128827    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerName="extract"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128832    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerName="extract"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128842    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eebdc326-181f-4562-8a32-4b054c249218" containerName="util"
Sep 09 06:58:27.128874 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128847    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eebdc326-181f-4562-8a32-4b054c249218" containerName="util"
Sep 09 06:58:27.129641 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128893    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eebdc326-181f-4562-8a32-4b054c249218" containerName="extract"
Sep 09 06:58:27.129641 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128901    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3296c90b-f0d9-4761-9719-de5ada54109f" containerName="extract"
Sep 09 06:58:27.129641 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128907    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="428a680f-e622-4cae-bbca-d1cc58b6d1b9" containerName="extract"
Sep 09 06:58:27.129641 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.128914    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bfe18d6-03d1-45eb-98eb-4659ed5b6734" containerName="extract"
Sep 09 06:58:27.131295 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.131279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:27.135215 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.135192    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 06:58:27.135345 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.135193    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 06:58:27.135345 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.135310    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-z5v8j\""
Sep 09 06:58:27.145856 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.145833    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"]
Sep 09 06:58:27.168699 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.168373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbv89\" (UniqueName: \"kubernetes.io/projected/3addc60c-f8fb-40bc-8e69-f191865cbf40-kube-api-access-fbv89\") pod \"limitador-operator-controller-manager-74495c69f8-v9g5p\" (UID: \"3addc60c-f8fb-40bc-8e69-f191865cbf40\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:27.268821 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.268789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbv89\" (UniqueName: \"kubernetes.io/projected/3addc60c-f8fb-40bc-8e69-f191865cbf40-kube-api-access-fbv89\") pod \"limitador-operator-controller-manager-74495c69f8-v9g5p\" (UID: \"3addc60c-f8fb-40bc-8e69-f191865cbf40\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:27.281226 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.281197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbv89\" (UniqueName: \"kubernetes.io/projected/3addc60c-f8fb-40bc-8e69-f191865cbf40-kube-api-access-fbv89\") pod \"limitador-operator-controller-manager-74495c69f8-v9g5p\" (UID: \"3addc60c-f8fb-40bc-8e69-f191865cbf40\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:27.440738 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.440707    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:27.713268 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:27.713244    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"]
Sep 09 06:58:27.715181 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:58:27.715153    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3addc60c_f8fb_40bc_8e69_f191865cbf40.slice/crio-3e262c6da7cc478bd48113d762dcea8028b25af03fcbe5f62082c6dadfe2cb0a WatchSource:0}: Error finding container 3e262c6da7cc478bd48113d762dcea8028b25af03fcbe5f62082c6dadfe2cb0a: Status 404 returned error can't find the container with id 3e262c6da7cc478bd48113d762dcea8028b25af03fcbe5f62082c6dadfe2cb0a
Sep 09 06:58:28.199239 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:28.199199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p" event={"ID":"3addc60c-f8fb-40bc-8e69-f191865cbf40","Type":"ContainerStarted","Data":"3e262c6da7cc478bd48113d762dcea8028b25af03fcbe5f62082c6dadfe2cb0a"}
Sep 09 06:58:31.215975 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:31.215937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p" event={"ID":"3addc60c-f8fb-40bc-8e69-f191865cbf40","Type":"ContainerStarted","Data":"5bab88b003a9b77bee189167868ac76bf9d04023f4ccc3c5f0c42e1ef9c89cac"}
Sep 09 06:58:31.216400 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:31.216075    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:31.243285 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:31.243233    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p" podStartSLOduration=1.491156578 podStartE2EDuration="4.243218264s" totalImagesPullingTime="2.752061686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:58:31.241485419 +0000 UTC m=+434.461726532" watchObservedRunningTime="2026-09-09 06:58:31.243218264 +0000 UTC m=+434.463459377"
Sep 09 06:58:42.224007 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:42.223929    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:43.727332 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.727298    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"]
Sep 09 06:58:43.729604 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.729582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.732168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.732141    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 06:58:43.732168 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.732160    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 06:58:43.732352 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.732196    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ssn7s\""
Sep 09 06:58:43.738246 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.738224    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"]
Sep 09 06:58:43.802269 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.802231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4sgn\" (UniqueName: \"kubernetes.io/projected/ed21057c-9582-46c7-9922-e5e4267ec8f1-kube-api-access-r4sgn\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.802436 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.802305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed21057c-9582-46c7-9922-e5e4267ec8f1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.802436 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.802328    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed21057c-9582-46c7-9922-e5e4267ec8f1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.903544 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.903514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed21057c-9582-46c7-9922-e5e4267ec8f1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.903544 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.903546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed21057c-9582-46c7-9922-e5e4267ec8f1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.903791 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.903578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4sgn\" (UniqueName: \"kubernetes.io/projected/ed21057c-9582-46c7-9922-e5e4267ec8f1-kube-api-access-r4sgn\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.904139 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.904096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed21057c-9582-46c7-9922-e5e4267ec8f1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.906062 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.906039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed21057c-9582-46c7-9922-e5e4267ec8f1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:43.911845 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:43.911821    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4sgn\" (UniqueName: \"kubernetes.io/projected/ed21057c-9582-46c7-9922-e5e4267ec8f1-kube-api-access-r4sgn\") pod \"kuadrant-console-plugin-fbdd4f445-drgw7\" (UID: \"ed21057c-9582-46c7-9922-e5e4267ec8f1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:44.039590 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:44.039506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"
Sep 09 06:58:44.309230 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:44.309163    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7"]
Sep 09 06:58:44.311386 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:58:44.311355    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded21057c_9582_46c7_9922_e5e4267ec8f1.slice/crio-fc45458b361c5a39b742954601c69bc181df1d08ff4bb5218144a971f6ba6466 WatchSource:0}: Error finding container fc45458b361c5a39b742954601c69bc181df1d08ff4bb5218144a971f6ba6466: Status 404 returned error can't find the container with id fc45458b361c5a39b742954601c69bc181df1d08ff4bb5218144a971f6ba6466
Sep 09 06:58:45.282859 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:45.282821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7" event={"ID":"ed21057c-9582-46c7-9922-e5e4267ec8f1","Type":"ContainerStarted","Data":"fc45458b361c5a39b742954601c69bc181df1d08ff4bb5218144a971f6ba6466"}
Sep 09 06:58:54.816197 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.816091    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"]
Sep 09 06:58:54.816758 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.816389    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p" podUID="3addc60c-f8fb-40bc-8e69-f191865cbf40" containerName="manager" containerID="cri-o://5bab88b003a9b77bee189167868ac76bf9d04023f4ccc3c5f0c42e1ef9c89cac" gracePeriod=2
Sep 09 06:58:54.842882 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.842850    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"]
Sep 09 06:58:54.848315 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:58:54.848279    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3addc60c-f8fb-40bc-8e69-f191865cbf40\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-v9g5p\": pods \"limitador-operator-controller-manager-74495c69f8-v9g5p\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:54.851988 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.851966    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"]
Sep 09 06:58:54.852423 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.852404    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3addc60c-f8fb-40bc-8e69-f191865cbf40" containerName="manager"
Sep 09 06:58:54.852534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.852425    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3addc60c-f8fb-40bc-8e69-f191865cbf40" containerName="manager"
Sep 09 06:58:54.852534 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.852528    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3addc60c-f8fb-40bc-8e69-f191865cbf40" containerName="manager"
Sep 09 06:58:54.854917 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.854900    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"
Sep 09 06:58:54.857453 ip-10-0-142-122 kubenswrapper[2562]: E0909 06:58:54.857422    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-v9g5p\" is forbidden: User \"system:node:ip-10-0-142-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-122.ec2.internal' and this object" podUID="3addc60c-f8fb-40bc-8e69-f191865cbf40" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:54.877702 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.877672    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"]
Sep 09 06:58:54.996648 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:54.996619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6rr\" (UniqueName: \"kubernetes.io/projected/5d007b14-48c5-4efb-a945-dd0f1b73b736-kube-api-access-zs6rr\") pod \"limitador-operator-controller-manager-74495c69f8-n4b4j\" (UID: \"5d007b14-48c5-4efb-a945-dd0f1b73b736\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"
Sep 09 06:58:55.097907 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:55.097823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6rr\" (UniqueName: \"kubernetes.io/projected/5d007b14-48c5-4efb-a945-dd0f1b73b736-kube-api-access-zs6rr\") pod \"limitador-operator-controller-manager-74495c69f8-n4b4j\" (UID: \"5d007b14-48c5-4efb-a945-dd0f1b73b736\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"
Sep 09 06:58:55.113248 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:55.113214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6rr\" (UniqueName: \"kubernetes.io/projected/5d007b14-48c5-4efb-a945-dd0f1b73b736-kube-api-access-zs6rr\") pod \"limitador-operator-controller-manager-74495c69f8-n4b4j\" (UID: \"5d007b14-48c5-4efb-a945-dd0f1b73b736\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"
Sep 09 06:58:55.188822 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:55.188780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"
Sep 09 06:58:55.335421 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:55.335380    2562 generic.go:356] "Generic (PLEG): container finished" podID="3addc60c-f8fb-40bc-8e69-f191865cbf40" containerID="5bab88b003a9b77bee189167868ac76bf9d04023f4ccc3c5f0c42e1ef9c89cac" exitCode=0
Sep 09 06:58:56.579670 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:56.579645    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:56.712894 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:56.712858    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbv89\" (UniqueName: \"kubernetes.io/projected/3addc60c-f8fb-40bc-8e69-f191865cbf40-kube-api-access-fbv89\") pod \"3addc60c-f8fb-40bc-8e69-f191865cbf40\" (UID: \"3addc60c-f8fb-40bc-8e69-f191865cbf40\") "
Sep 09 06:58:56.715384 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:56.715348    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3addc60c-f8fb-40bc-8e69-f191865cbf40-kube-api-access-fbv89" (OuterVolumeSpecName: "kube-api-access-fbv89") pod "3addc60c-f8fb-40bc-8e69-f191865cbf40" (UID: "3addc60c-f8fb-40bc-8e69-f191865cbf40"). InnerVolumeSpecName "kube-api-access-fbv89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 06:58:56.814471 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:56.814379    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbv89\" (UniqueName: \"kubernetes.io/projected/3addc60c-f8fb-40bc-8e69-f191865cbf40-kube-api-access-fbv89\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 06:58:57.231365 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:57.231331    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3addc60c-f8fb-40bc-8e69-f191865cbf40" path="/var/lib/kubelet/pods/3addc60c-f8fb-40bc-8e69-f191865cbf40/volumes"
Sep 09 06:58:57.348017 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:57.347985    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-v9g5p"
Sep 09 06:58:57.348197 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:57.348030    2562 scope.go:117] "RemoveContainer" containerID="5bab88b003a9b77bee189167868ac76bf9d04023f4ccc3c5f0c42e1ef9c89cac"
Sep 09 06:58:57.671723 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:57.671695    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"]
Sep 09 06:58:57.673958 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:58:57.673916    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d007b14_48c5_4efb_a945_dd0f1b73b736.slice/crio-05605afc63468ff2d727b6b7aa910df2eee1ae946b337b4bf5b651b92d4a2a35 WatchSource:0}: Error finding container 05605afc63468ff2d727b6b7aa910df2eee1ae946b337b4bf5b651b92d4a2a35: Status 404 returned error can't find the container with id 05605afc63468ff2d727b6b7aa910df2eee1ae946b337b4bf5b651b92d4a2a35
Sep 09 06:58:58.353637 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:58.353599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7" event={"ID":"ed21057c-9582-46c7-9922-e5e4267ec8f1","Type":"ContainerStarted","Data":"0538ba605c817ced99391c63bd028252d54266d69e7162cbd3243fbcb1f1bf89"}
Sep 09 06:58:58.355334 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:58.355307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j" event={"ID":"5d007b14-48c5-4efb-a945-dd0f1b73b736","Type":"ContainerStarted","Data":"c98b33e80d2d8b3f9ed647a46ec029dab1d9b43b8715cd78ce40ece840f484d9"}
Sep 09 06:58:58.355486 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:58.355355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j" event={"ID":"5d007b14-48c5-4efb-a945-dd0f1b73b736","Type":"ContainerStarted","Data":"05605afc63468ff2d727b6b7aa910df2eee1ae946b337b4bf5b651b92d4a2a35"}
Sep 09 06:58:58.355486 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:58.355390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"
Sep 09 06:58:58.372912 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:58.372865    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-drgw7" podStartSLOduration=2.245544484 podStartE2EDuration="15.372851001s" totalImagesPullingTime="13.127306517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:58:58.370715754 +0000 UTC m=+461.590956878" watchObservedRunningTime="2026-09-09 06:58:58.372851001 +0000 UTC m=+461.593092114"
Sep 09 06:58:58.389293 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:58:58.389255    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j" podStartSLOduration=4.389242107 podStartE2EDuration="4.389242107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:58:58.38825007 +0000 UTC m=+461.608491183" watchObservedRunningTime="2026-09-09 06:58:58.389242107 +0000 UTC m=+461.609483219"
Sep 09 06:59:09.363533 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:09.363505    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-n4b4j"
Sep 09 06:59:26.708610 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.708576    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"]
Sep 09 06:59:26.720901 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.720871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.723943 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.723825    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9gg97\""
Sep 09 06:59:26.725026 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.725002    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"]
Sep 09 06:59:26.739900 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.739873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.740022 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.739909    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de492c1-d3e3-4d28-a914-758056b4e6b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.740022 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.739940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.740022 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.739961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.740022 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.739978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.740251 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.740052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.740251 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.740079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.740251 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.740152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.740251 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.740187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47tvn\" (UniqueName: \"kubernetes.io/projected/9de492c1-d3e3-4d28-a914-758056b4e6b9-kube-api-access-47tvn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841091 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841060    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841091 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de492c1-d3e3-4d28-a914-758056b4e6b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841337 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841337 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841337 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841337 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841563 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841563 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841563 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47tvn\" (UniqueName: \"kubernetes.io/projected/9de492c1-d3e3-4d28-a914-758056b4e6b9-kube-api-access-47tvn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841563 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841537    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841762 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841762 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841762 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.841931 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.841912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de492c1-d3e3-4d28-a914-758056b4e6b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.843704 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.843682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.844012 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.843995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.849573 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.849550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de492c1-d3e3-4d28-a914-758056b4e6b9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:26.849692 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:26.849673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47tvn\" (UniqueName: \"kubernetes.io/projected/9de492c1-d3e3-4d28-a914-758056b4e6b9-kube-api-access-47tvn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9r9cc\" (UID: \"9de492c1-d3e3-4d28-a914-758056b4e6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:27.033320 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:27.033236    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:27.385767 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:27.385737    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"]
Sep 09 06:59:27.386749 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:59:27.386719    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de492c1_d3e3_4d28_a914_758056b4e6b9.slice/crio-d0f364f337d8819ba9ae0b189a9d5da40dade61498f421f011588a383382829e WatchSource:0}: Error finding container d0f364f337d8819ba9ae0b189a9d5da40dade61498f421f011588a383382829e: Status 404 returned error can't find the container with id d0f364f337d8819ba9ae0b189a9d5da40dade61498f421f011588a383382829e
Sep 09 06:59:27.388941 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:27.388896    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 06:59:27.389021 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:27.388987    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 06:59:27.389062 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:27.389028    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 06:59:27.499222 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:27.499188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" event={"ID":"9de492c1-d3e3-4d28-a914-758056b4e6b9","Type":"ContainerStarted","Data":"d0f364f337d8819ba9ae0b189a9d5da40dade61498f421f011588a383382829e"}
Sep 09 06:59:28.505332 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:28.505292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" event={"ID":"9de492c1-d3e3-4d28-a914-758056b4e6b9","Type":"ContainerStarted","Data":"9adf58cd15254e436575a2c0e965d90428dddd7d424484b57882da1baaf53ee7"}
Sep 09 06:59:28.529364 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:28.529308    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" podStartSLOduration=2.529290211 podStartE2EDuration="2.529290211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:59:28.527144851 +0000 UTC m=+491.747385966" watchObservedRunningTime="2026-09-09 06:59:28.529290211 +0000 UTC m=+491.749531325"
Sep 09 06:59:29.033606 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:29.033571    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:29.038185 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:29.038157    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:29.510202 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:29.510171    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:29.511247 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:29.511225    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 06:59:41.633212 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.633176    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 06:59:41.635961 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.635941    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:41.638268 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.638244    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 06:59:41.648067 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.648042    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 06:59:41.660364 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.660339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b83b2514-b189-424c-93ac-2e4b63b0c669-config-file\") pod \"limitador-limitador-84994f7ffc-lv5s8\" (UID: \"b83b2514-b189-424c-93ac-2e4b63b0c669\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:41.660468 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.660409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdkp\" (UniqueName: \"kubernetes.io/projected/b83b2514-b189-424c-93ac-2e4b63b0c669-kube-api-access-9jdkp\") pod \"limitador-limitador-84994f7ffc-lv5s8\" (UID: \"b83b2514-b189-424c-93ac-2e4b63b0c669\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:41.679241 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.679217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 06:59:41.761826 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.761776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdkp\" (UniqueName: \"kubernetes.io/projected/b83b2514-b189-424c-93ac-2e4b63b0c669-kube-api-access-9jdkp\") pod \"limitador-limitador-84994f7ffc-lv5s8\" (UID: \"b83b2514-b189-424c-93ac-2e4b63b0c669\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:41.762015 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.761869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b83b2514-b189-424c-93ac-2e4b63b0c669-config-file\") pod \"limitador-limitador-84994f7ffc-lv5s8\" (UID: \"b83b2514-b189-424c-93ac-2e4b63b0c669\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:41.762537 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.762513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b83b2514-b189-424c-93ac-2e4b63b0c669-config-file\") pod \"limitador-limitador-84994f7ffc-lv5s8\" (UID: \"b83b2514-b189-424c-93ac-2e4b63b0c669\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:41.769702 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.769681    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdkp\" (UniqueName: \"kubernetes.io/projected/b83b2514-b189-424c-93ac-2e4b63b0c669-kube-api-access-9jdkp\") pod \"limitador-limitador-84994f7ffc-lv5s8\" (UID: \"b83b2514-b189-424c-93ac-2e4b63b0c669\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:41.945640 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:41.945603    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:42.217461 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:42.217435    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 06:59:42.219683 ip-10-0-142-122 kubenswrapper[2562]: W0909 06:59:42.219655    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83b2514_b189_424c_93ac_2e4b63b0c669.slice/crio-30d6090006a1ec69246deb8acba44ceda11dc24e1867d3c8482859102e4044be WatchSource:0}: Error finding container 30d6090006a1ec69246deb8acba44ceda11dc24e1867d3c8482859102e4044be: Status 404 returned error can't find the container with id 30d6090006a1ec69246deb8acba44ceda11dc24e1867d3c8482859102e4044be
Sep 09 06:59:42.574483 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:42.574395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8" event={"ID":"b83b2514-b189-424c-93ac-2e4b63b0c669","Type":"ContainerStarted","Data":"30d6090006a1ec69246deb8acba44ceda11dc24e1867d3c8482859102e4044be"}
Sep 09 06:59:43.580732 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:43.580691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8" event={"ID":"b83b2514-b189-424c-93ac-2e4b63b0c669","Type":"ContainerStarted","Data":"1982bf142b4c5b724abe9b29ad77ede4a011395a0695ce9ea6e3d7a0343c1dc4"}
Sep 09 06:59:43.581119 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:43.580791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 06:59:43.598001 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:43.597951    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8" podStartSLOduration=1.69304034 podStartE2EDuration="2.597934207s" totalImagesPullingTime="904.893867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 06:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 06:59:43.597323699 +0000 UTC m=+506.817564836" watchObservedRunningTime="2026-09-09 06:59:43.597934207 +0000 UTC m=+506.818175335"
Sep 09 06:59:54.586756 ip-10-0-142-122 kubenswrapper[2562]: I0909 06:59:54.586720    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"
Sep 09 07:00:54.447280 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:00:54.447242    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:01:03.966745 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:03.966709    2562 generic.go:356] "Generic (PLEG): container finished" podID="9de492c1-d3e3-4d28-a914-758056b4e6b9" containerID="9adf58cd15254e436575a2c0e965d90428dddd7d424484b57882da1baaf53ee7" exitCode=0
Sep 09 07:01:03.966745 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:03.966749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" event={"ID":"9de492c1-d3e3-4d28-a914-758056b4e6b9","Type":"ContainerDied","Data":"9adf58cd15254e436575a2c0e965d90428dddd7d424484b57882da1baaf53ee7"}
Sep 09 07:01:03.967243 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:03.967134    2562 scope.go:117] "RemoveContainer" containerID="9adf58cd15254e436575a2c0e965d90428dddd7d424484b57882da1baaf53ee7"
Sep 09 07:01:03.968122 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:03.968075    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:01:03.968181 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:03.968170    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:01:03.968225 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:03.968204    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:01:04.973030 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:04.972996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" event={"ID":"9de492c1-d3e3-4d28-a914-758056b4e6b9","Type":"ContainerStarted","Data":"741de12224524b76c9573da718951b827976c6372cae54bf05e5cd395f6f7d5b"}
Sep 09 07:01:05.033394 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:05.033352    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 07:01:05.033640 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:05.033618    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9r9cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 07:01:05.033692 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:05.033668    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" podUID="9de492c1-d3e3-4d28-a914-758056b4e6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 07:01:06.034417 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:06.034380    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9r9cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 07:01:06.034805 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:06.034433    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" podUID="9de492c1-d3e3-4d28-a914-758056b4e6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 07:01:07.034237 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:07.034204    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9r9cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 07:01:07.034404 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:07.034268    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" podUID="9de492c1-d3e3-4d28-a914-758056b4e6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 07:01:08.034498 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:08.034461    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9r9cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 07:01:08.034894 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:08.034530    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" podUID="9de492c1-d3e3-4d28-a914-758056b4e6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 07:01:09.034261 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:09.034225    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9r9cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 07:01:09.034441 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:09.034284    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" podUID="9de492c1-d3e3-4d28-a914-758056b4e6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 07:01:10.034501 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:10.034462    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9r9cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Sep 09 07:01:10.034963 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:10.034520    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc" podUID="9de492c1-d3e3-4d28-a914-758056b4e6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Sep 09 07:01:11.037781 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:11.037752    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 07:01:11.038292 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:11.038121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 07:01:11.038765 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:11.038747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9r9cc"
Sep 09 07:01:17.152919 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:17.152892    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:01:17.155240 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:17.155212    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:01:23.761374 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:23.761339    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:01:30.065770 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:30.065730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:01:36.463063 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:36.463026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:01:43.471581 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:43.471537    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:01:52.324895 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.324857    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"]
Sep 09 07:01:52.327477 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.327459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.330973 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.330942    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 07:01:52.330973 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.330966    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nnc67\""
Sep 09 07:01:52.331193 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.330950    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 07:01:52.331193 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.330974    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 07:01:52.340252 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.340231    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"]
Sep 09 07:01:52.472866 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.472832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.472866 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.472866    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.473064 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.472891    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.473064 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.472977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq46s\" (UniqueName: \"kubernetes.io/projected/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-kube-api-access-fq46s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.473064 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.473025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.473208 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.473075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.573957 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.573918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.574153 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.573962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.574153 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.574082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.574281 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.574167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq46s\" (UniqueName: \"kubernetes.io/projected/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-kube-api-access-fq46s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.574281 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.574231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.574281 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.574260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.574486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.574467    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.574561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.574543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.574646 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.574625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.576297 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.576243    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.576869 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.576734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.583983 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.583960    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq46s\" (UniqueName: \"kubernetes.io/projected/0a3aa76f-bb1d-4236-918b-bce76c4f97d2-kube-api-access-fq46s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g\" (UID: \"0a3aa76f-bb1d-4236-918b-bce76c4f97d2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.636991 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.636959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:01:52.907928 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.907901    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"]
Sep 09 07:01:52.910130 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:01:52.910091    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3aa76f_bb1d_4236_918b_bce76c4f97d2.slice/crio-b12e97b9d59b5e5d274b634d8863b3f513e54ad1d5b8604a18f7b599e56b3727 WatchSource:0}: Error finding container b12e97b9d59b5e5d274b634d8863b3f513e54ad1d5b8604a18f7b599e56b3727: Status 404 returned error can't find the container with id b12e97b9d59b5e5d274b634d8863b3f513e54ad1d5b8604a18f7b599e56b3727
Sep 09 07:01:52.911328 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:52.911312    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 07:01:53.220387 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:53.220347    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g" event={"ID":"0a3aa76f-bb1d-4236-918b-bce76c4f97d2","Type":"ContainerStarted","Data":"b12e97b9d59b5e5d274b634d8863b3f513e54ad1d5b8604a18f7b599e56b3727"}
Sep 09 07:01:54.056545 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:54.056514    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:01:54.226796 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:54.226761    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g" event={"ID":"0a3aa76f-bb1d-4236-918b-bce76c4f97d2","Type":"ContainerStarted","Data":"9b6ab2ecbdfa6597f60987f38d70e67520c431bb1a4b9c0005ea3047eccf985c"}
Sep 09 07:01:54.248811 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:01:54.248758    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g" podStartSLOduration=1.314109446 podStartE2EDuration="2.248741256s" totalImagesPullingTime="934.63181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:01:54.247078854 +0000 UTC m=+637.467319968" watchObservedRunningTime="2026-09-09 07:01:54.248741256 +0000 UTC m=+637.468982421"
Sep 09 07:02:02.637221 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:02.637180    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:02:02.637652 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:02.637251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:02:02.650079 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:02.650046    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:02:03.074189 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:03.074140    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:02:03.286020 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:03.285991    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g"
Sep 09 07:02:24.429122 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.429081    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"]
Sep 09 07:02:24.431331 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.431311    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.433848 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.433823    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 07:02:24.442748 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.442728    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"]
Sep 09 07:02:24.558551 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.558517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.558551 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.558553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257b4\" (UniqueName: \"kubernetes.io/projected/5af1ef7d-3e54-4354-8fa2-4146edd70968-kube-api-access-257b4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.558765 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.558588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af1ef7d-3e54-4354-8fa2-4146edd70968-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.558765 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.558651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.558765 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.558697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.558765 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.558716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.659707 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.659672    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.659898 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.659715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.659898 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.659736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.659898 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.659766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.659898 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.659787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257b4\" (UniqueName: \"kubernetes.io/projected/5af1ef7d-3e54-4354-8fa2-4146edd70968-kube-api-access-257b4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.659898 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.659822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af1ef7d-3e54-4354-8fa2-4146edd70968-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.660207 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.660152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.660256 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.660219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.660291 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.660255    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.662045 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.662017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5af1ef7d-3e54-4354-8fa2-4146edd70968-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.662318 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.662302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af1ef7d-3e54-4354-8fa2-4146edd70968-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.668367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.668343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-257b4\" (UniqueName: \"kubernetes.io/projected/5af1ef7d-3e54-4354-8fa2-4146edd70968-kube-api-access-257b4\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx\" (UID: \"5af1ef7d-3e54-4354-8fa2-4146edd70968\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:24.740675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:24.740603    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:25.011063 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:25.011037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"]
Sep 09 07:02:25.013288 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:02:25.013251    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5af1ef7d_3e54_4354_8fa2_4146edd70968.slice/crio-46e8ee779a995ab27304c498a6df50fed453fa50505b1229db39b53e6fd23cf2 WatchSource:0}: Error finding container 46e8ee779a995ab27304c498a6df50fed453fa50505b1229db39b53e6fd23cf2: Status 404 returned error can't find the container with id 46e8ee779a995ab27304c498a6df50fed453fa50505b1229db39b53e6fd23cf2
Sep 09 07:02:25.395227 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:25.395133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx" event={"ID":"5af1ef7d-3e54-4354-8fa2-4146edd70968","Type":"ContainerStarted","Data":"8b1911b5128f02bab7572853672b642fa8d58cec7abf27b470893ede3c5688a8"}
Sep 09 07:02:25.395227 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:25.395180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx" event={"ID":"5af1ef7d-3e54-4354-8fa2-4146edd70968","Type":"ContainerStarted","Data":"46e8ee779a995ab27304c498a6df50fed453fa50505b1229db39b53e6fd23cf2"}
Sep 09 07:02:25.414568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:25.414518    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx" podStartSLOduration=1.229918713 podStartE2EDuration="1.414501438s" totalImagesPullingTime="184.582725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:02:25.413644764 +0000 UTC m=+668.633885877" watchObservedRunningTime="2026-09-09 07:02:25.414501438 +0000 UTC m=+668.634742550"
Sep 09 07:02:26.466471 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:26.466436    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:02:34.741306 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:34.741271    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:34.741710 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:34.741319    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:34.753623 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:34.753598    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:02:35.455399 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:02:35.455373    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx"
Sep 09 07:04:07.643146 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.643093    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"]
Sep 09 07:04:07.646909 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.646887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.649324 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.649304    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b212cb-openshift-default-dockercfg-fxsgw\""
Sep 09 07:04:07.667236 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.667210    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"]
Sep 09 07:04:07.739833 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.739800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-socket\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.740002 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.739846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-credential-socket\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.740002 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.739876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004450e9-6a47-4f78-9b14-f355b707fd28-istio-podinfo\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.740002 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.739904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-certs\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.740002 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.739935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-data\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.740002 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.739958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-istio-token\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.740002 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.739987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptkt\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-kube-api-access-dptkt\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.740255 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.740043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004450e9-6a47-4f78-9b14-f355b707fd28-istiod-ca-cert\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.740255 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.740085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-envoy\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841017 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.840973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-envoy\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-socket\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-credential-socket\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004450e9-6a47-4f78-9b14-f355b707fd28-istio-podinfo\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-certs\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-data\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-istio-token\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841536 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptkt\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-kube-api-access-dptkt\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841536 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004450e9-6a47-4f78-9b14-f355b707fd28-istiod-ca-cert\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841536 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841499    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-socket\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841813 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841772    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-data\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841961 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-credential-socket\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.841961 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.841930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-certs\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.842072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.842054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004450e9-6a47-4f78-9b14-f355b707fd28-istiod-ca-cert\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.843489 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.843469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-envoy\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.844155 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.844134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004450e9-6a47-4f78-9b14-f355b707fd28-istio-podinfo\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.850319 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.850296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-istio-token\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.850611 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.850590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptkt\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-kube-api-access-dptkt\") pod \"e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") " pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:07.957934 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:07.957901    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:08.232854 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:08.232789    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"]
Sep 09 07:04:08.235601 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:04:08.235570    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004450e9_6a47_4f78_9b14_f355b707fd28.slice/crio-cce5491f977d8f27fda2ff669be67f3513d80ef5d802c017d9fdd9bf2be43256 WatchSource:0}: Error finding container cce5491f977d8f27fda2ff669be67f3513d80ef5d802c017d9fdd9bf2be43256: Status 404 returned error can't find the container with id cce5491f977d8f27fda2ff669be67f3513d80ef5d802c017d9fdd9bf2be43256
Sep 09 07:04:08.237699 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:08.237664    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:08.237768 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:08.237738    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:08.237806 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:08.237771    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:08.926241 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:08.926209    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" event={"ID":"004450e9-6a47-4f78-9b14-f355b707fd28","Type":"ContainerStarted","Data":"d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3"}
Sep 09 07:04:08.926684 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:08.926263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" event={"ID":"004450e9-6a47-4f78-9b14-f355b707fd28","Type":"ContainerStarted","Data":"cce5491f977d8f27fda2ff669be67f3513d80ef5d802c017d9fdd9bf2be43256"}
Sep 09 07:04:08.948043 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:08.947995    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podStartSLOduration=1.947978207 podStartE2EDuration="1.947978207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:04:08.94703186 +0000 UTC m=+772.167272973" watchObservedRunningTime="2026-09-09 07:04:08.947978207 +0000 UTC m=+772.168219319"
Sep 09 07:04:08.958235 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:08.958208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:09.963591 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:09.963556    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:10.568007 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:10.567970    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:04:10.938117 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:10.938076    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:10.939053 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:10.939035    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:14.425752 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:14.425711    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7955-140037892555952"]
Sep 09 07:04:14.427856 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:14.427838    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:14.430120 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:14.430086    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dw24h\""
Sep 09 07:04:14.433688 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:14.433667    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7955-140037892555952"]
Sep 09 07:04:14.503692 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:14.503657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd4c\" (UniqueName: \"kubernetes.io/projected/27c97bde-352a-4671-9aea-4cea2551e5ff-kube-api-access-dwd4c\") pod \"test-curl-7955-140037892555952\" (UID: \"27c97bde-352a-4671-9aea-4cea2551e5ff\") " pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:14.605035 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:14.605001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd4c\" (UniqueName: \"kubernetes.io/projected/27c97bde-352a-4671-9aea-4cea2551e5ff-kube-api-access-dwd4c\") pod \"test-curl-7955-140037892555952\" (UID: \"27c97bde-352a-4671-9aea-4cea2551e5ff\") " pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:14.612927 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:14.612903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd4c\" (UniqueName: \"kubernetes.io/projected/27c97bde-352a-4671-9aea-4cea2551e5ff-kube-api-access-dwd4c\") pod \"test-curl-7955-140037892555952\" (UID: \"27c97bde-352a-4671-9aea-4cea2551e5ff\") " pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:14.738192 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:14.738091    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:15.006512 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.006474    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7955-140037892555952"]
Sep 09 07:04:15.007706 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:04:15.007677    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27c97bde_352a_4671_9aea_4cea2551e5ff.slice/crio-688e3f51aa13881e728b2566100da98f11200de677a6a6bc4011ed311d791545 WatchSource:0}: Error finding container 688e3f51aa13881e728b2566100da98f11200de677a6a6bc4011ed311d791545: Status 404 returned error can't find the container with id 688e3f51aa13881e728b2566100da98f11200de677a6a6bc4011ed311d791545
Sep 09 07:04:15.601201 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.601166    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"]
Sep 09 07:04:15.604416 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.604396    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.606863 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.606837    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ef07c382-openshift-default-dockercfg-t6724\""
Sep 09 07:04:15.617546 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.617522    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"]
Sep 09 07:04:15.717066 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-podinfo\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.717066 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istiod-ca-cert\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.717412 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmj6\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-kube-api-access-pxmj6\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.717412 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-credential-socket\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.717412 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-socket\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.717412 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-envoy\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.717412 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-certs\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.717663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-data\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.717663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.717473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-token\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.818940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.818610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-certs\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.818940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.818679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-data\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.818940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.818728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-token\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.818940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.818764    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-podinfo\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.818940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.818788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istiod-ca-cert\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.818940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.818836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmj6\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-kube-api-access-pxmj6\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.818940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.818864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-credential-socket\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.818940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.818901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-socket\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.820447 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.820417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-envoy\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.820575 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.820471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istiod-ca-cert\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.820747 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.820726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-credential-socket\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.820978 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.820939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-data\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.820978 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.820966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-socket\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.821269 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.821248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-certs\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.823553 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.823527    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-podinfo\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.823800 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.823780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-envoy\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.830712 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.830689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-token\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.831097 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.831073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmj6\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-kube-api-access-pxmj6\") pod \"e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") " pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.916680 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.916643    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:15.968410 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:15.968372    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7955-140037892555952" event={"ID":"27c97bde-352a-4671-9aea-4cea2551e5ff","Type":"ContainerStarted","Data":"688e3f51aa13881e728b2566100da98f11200de677a6a6bc4011ed311d791545"}
Sep 09 07:04:16.205370 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:16.205336    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"]
Sep 09 07:04:16.208318 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:04:16.208280    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8c2d4f_3de5_4d05_94f4_48902a2e1477.slice/crio-24dc41b8fd641731e915cec6583cf24c4c0af1bea0449c1164e790948c244a8b WatchSource:0}: Error finding container 24dc41b8fd641731e915cec6583cf24c4c0af1bea0449c1164e790948c244a8b: Status 404 returned error can't find the container with id 24dc41b8fd641731e915cec6583cf24c4c0af1bea0449c1164e790948c244a8b
Sep 09 07:04:16.211521 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:16.211479    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:16.211611 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:16.211567    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:16.211663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:16.211611    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:16.975970 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:16.975938    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" event={"ID":"0d8c2d4f-3de5-4d05-94f4-48902a2e1477","Type":"ContainerStarted","Data":"9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312"}
Sep 09 07:04:16.976429 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:16.975999    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" event={"ID":"0d8c2d4f-3de5-4d05-94f4-48902a2e1477","Type":"ContainerStarted","Data":"24dc41b8fd641731e915cec6583cf24c4c0af1bea0449c1164e790948c244a8b"}
Sep 09 07:04:17.001259 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:17.001212    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" podStartSLOduration=2.001191253 podStartE2EDuration="2.001191253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:04:16.998922745 +0000 UTC m=+780.219163858" watchObservedRunningTime="2026-09-09 07:04:17.001191253 +0000 UTC m=+780.221432383"
Sep 09 07:04:17.917754 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:17.917715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:17.923184 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:17.923154    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:17.985787 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:17.985749    2562 generic.go:356] "Generic (PLEG): container finished" podID="27c97bde-352a-4671-9aea-4cea2551e5ff" containerID="aec8f948f80fbe9a271d1780f5e2ac3e3ad7591f92616094dfc072e116fb5e71" exitCode=0
Sep 09 07:04:17.987759 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:17.987729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7955-140037892555952" event={"ID":"27c97bde-352a-4671-9aea-4cea2551e5ff","Type":"ContainerDied","Data":"aec8f948f80fbe9a271d1780f5e2ac3e3ad7591f92616094dfc072e116fb5e71"}
Sep 09 07:04:17.988235 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:17.988217    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:17.989562 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:17.989538    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:19.140719 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.140695    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:19.194825 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.194792    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7955-140037892555952_27c97bde-352a-4671-9aea-4cea2551e5ff/test-curl-7955-140037892555952/0.log"
Sep 09 07:04:19.231589 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.231565    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7955-140037892555952"]
Sep 09 07:04:19.231739 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.231601    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7955-140037892555952"]
Sep 09 07:04:19.253852 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.253790    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwd4c\" (UniqueName: \"kubernetes.io/projected/27c97bde-352a-4671-9aea-4cea2551e5ff-kube-api-access-dwd4c\") pod \"27c97bde-352a-4671-9aea-4cea2551e5ff\" (UID: \"27c97bde-352a-4671-9aea-4cea2551e5ff\") "
Sep 09 07:04:19.255869 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.255847    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c97bde-352a-4671-9aea-4cea2551e5ff-kube-api-access-dwd4c" (OuterVolumeSpecName: "kube-api-access-dwd4c") pod "27c97bde-352a-4671-9aea-4cea2551e5ff" (UID: "27c97bde-352a-4671-9aea-4cea2551e5ff"). InnerVolumeSpecName "kube-api-access-dwd4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:04:19.355478 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.355442    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwd4c\" (UniqueName: \"kubernetes.io/projected/27c97bde-352a-4671-9aea-4cea2551e5ff-kube-api-access-dwd4c\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:04:19.999428 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.999397    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="688e3f51aa13881e728b2566100da98f11200de677a6a6bc4011ed311d791545"
Sep 09 07:04:19.999428 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:19.999415    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:20.010387 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:04:20.010358    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7955-140037892555952\" is forbidden: User \"system:node:ip-10-0-142-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-122.ec2.internal' and this object" podUID="27c97bde-352a-4671-9aea-4cea2551e5ff" pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:20.042434 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.042404    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"]
Sep 09 07:04:20.042938 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.042920    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c97bde-352a-4671-9aea-4cea2551e5ff" containerName="test-curl-7955-140037892555952"
Sep 09 07:04:20.042938 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.042938    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c97bde-352a-4671-9aea-4cea2551e5ff" containerName="test-curl-7955-140037892555952"
Sep 09 07:04:20.043066 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.043026    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c97bde-352a-4671-9aea-4cea2551e5ff" containerName="test-curl-7955-140037892555952"
Sep 09 07:04:20.045373 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.045355    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.048246 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.048221    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-40c98b-openshift-default-dockercfg-m7s7h\""
Sep 09 07:04:20.059129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.059092    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"]
Sep 09 07:04:20.096494 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:04:20.096450    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7955-140037892555952\" is forbidden: User \"system:node:ip-10-0-142-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-122.ec2.internal' and this object" podUID="27c97bde-352a-4671-9aea-4cea2551e5ff" pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:20.162792 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.162761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppsdm\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-kube-api-access-ppsdm\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.162792 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.162795    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-certs\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.163395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.162824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-credential-socket\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.163395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.162845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-envoy\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.163395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.163020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-socket\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.163395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.163045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-data\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.163395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.163091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-podinfo\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.163395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.163162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-token\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.163395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.163193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istiod-ca-cert\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.182127 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.168684    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppsdm\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-kube-api-access-ppsdm\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-certs\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-credential-socket\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-envoy\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-socket\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-data\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-podinfo\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-token\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.272684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istiod-ca-cert\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.273506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-socket\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.273527    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istiod-ca-cert\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.274232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-credential-socket\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.274478    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-data\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.274663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.274571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-certs\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.280123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.276770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-envoy\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.281075 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.281051    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-podinfo\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.299686 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.287798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppsdm\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-kube-api-access-ppsdm\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.299686 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.291728    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-token\") pod \"e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") " pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.357040 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.357003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:20.655675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.655646    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"]
Sep 09 07:04:20.657186 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:04:20.657155    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c0aefc_35de_4f18_a689_09cd57f6a91e.slice/crio-6c33b3d72d4530f01d8caa6a35a6d67446e5ba75c7356733feeea7b6c413572c WatchSource:0}: Error finding container 6c33b3d72d4530f01d8caa6a35a6d67446e5ba75c7356733feeea7b6c413572c: Status 404 returned error can't find the container with id 6c33b3d72d4530f01d8caa6a35a6d67446e5ba75c7356733feeea7b6c413572c
Sep 09 07:04:20.659445 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.659411    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:20.659549 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.659492    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:20.659549 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:20.659538    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:21.006160 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:21.006096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" event={"ID":"c6c0aefc-35de-4f18-a689-09cd57f6a91e","Type":"ContainerStarted","Data":"ea1252d13363a1a9b0fc1511507e654d45ea9053616925c4be8175aa88aac5e4"}
Sep 09 07:04:21.006360 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:21.006173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" event={"ID":"c6c0aefc-35de-4f18-a689-09cd57f6a91e","Type":"ContainerStarted","Data":"6c33b3d72d4530f01d8caa6a35a6d67446e5ba75c7356733feeea7b6c413572c"}
Sep 09 07:04:21.031022 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:21.030967    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" podStartSLOduration=1.03095269 podStartE2EDuration="1.03095269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:04:21.029390256 +0000 UTC m=+784.249631363" watchObservedRunningTime="2026-09-09 07:04:21.03095269 +0000 UTC m=+784.251193838"
Sep 09 07:04:21.031336 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:04:21.031308    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7955-140037892555952\" is forbidden: User \"system:node:ip-10-0-142-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-122.ec2.internal' and this object" podUID="27c97bde-352a-4671-9aea-4cea2551e5ff" pod="opendatahub/test-curl-7955-140037892555952"
Sep 09 07:04:21.230401 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:21.230365    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c97bde-352a-4671-9aea-4cea2551e5ff" path="/var/lib/kubelet/pods/27c97bde-352a-4671-9aea-4cea2551e5ff/volumes"
Sep 09 07:04:21.358061 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:21.357967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:21.362724 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:21.362700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:22.011060 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:22.011026    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:22.011979 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:22.011956    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:26.509769 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:26.509555    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:51684->10.134.0.38:15021: read: connection reset by peer" start-of-body=
Sep 09 07:04:26.509769 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:26.509626    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:51684->10.134.0.38:15021: read: connection reset by peer"
Sep 09 07:04:27.037246 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:27.037212    2562 generic.go:356] "Generic (PLEG): container finished" podID="004450e9-6a47-4f78-9b14-f355b707fd28" containerID="d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3" exitCode=0
Sep 09 07:04:27.037424 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:27.037286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" event={"ID":"004450e9-6a47-4f78-9b14-f355b707fd28","Type":"ContainerDied","Data":"d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3"}
Sep 09 07:04:27.037706 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:27.037692    2562 scope.go:117] "RemoveContainer" containerID="d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3"
Sep 09 07:04:27.038760 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:27.038734    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:27.038817 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:27.038804    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:27.038852 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:27.038838    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:28.046737 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:28.046700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" event={"ID":"004450e9-6a47-4f78-9b14-f355b707fd28","Type":"ContainerStarted","Data":"24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a"}
Sep 09 07:04:28.958843 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:28.958808    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:28.958992 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:28.958933    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 07:04:28.958992 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:28.958967    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 07:04:29.958827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:29.958787    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 07:04:29.959231 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:29.958852    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 07:04:30.958766 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:30.958716    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 07:04:30.958924 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:30.958783    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 07:04:31.958868 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:31.958828    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 07:04:31.959038 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:31.958886    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 07:04:32.958913 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:32.958875    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 07:04:32.959099 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:32.958927    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 07:04:33.473841 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:33.473796    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:04:33.962333 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:33.962304    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:33.968129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:33.963397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:33.970031 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:33.970006    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:04:36.575068 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.575030    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"]
Sep 09 07:04:36.589186 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.589157    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.597841 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.597807    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"]
Sep 09 07:04:36.604248 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.599056    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fc9900-openshift-default-dockercfg-lrx4s\""
Sep 09 07:04:36.640356 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4xc\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-kube-api-access-tc4xc\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.640523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-socket\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.640523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a57d65-72e8-4306-9105-228095c56581-istiod-ca-cert\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.640523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-data\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.640523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-certs\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.640523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-credential-socket\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.640523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a57d65-72e8-4306-9105-228095c56581-istio-podinfo\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.640827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640534    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-envoy\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.640827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.640596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-istio-token\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.741616 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4xc\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-kube-api-access-tc4xc\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.741791 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-socket\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.741791 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a57d65-72e8-4306-9105-228095c56581-istiod-ca-cert\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.741791 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-data\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.741791 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-certs\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.741791 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-credential-socket\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.741791 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a57d65-72e8-4306-9105-228095c56581-istio-podinfo\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.742132 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-envoy\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.742132 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.741866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-istio-token\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.742132 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.742090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-socket\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.742425 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.742399    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-certs\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.742557 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.742432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-credential-socket\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.742557 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.742505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-data\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.742693 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.742673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a57d65-72e8-4306-9105-228095c56581-istiod-ca-cert\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.744315 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.744290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-envoy\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.744563 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.744547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a57d65-72e8-4306-9105-228095c56581-istio-podinfo\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.750540 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.750512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4xc\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-kube-api-access-tc4xc\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.751465 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.751443    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-istio-token\") pod \"e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") " pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:36.915086 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:36.915059    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:37.224308 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:37.224284    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"]
Sep 09 07:04:37.225993 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:04:37.225967    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a57d65_72e8_4306_9105_228095c56581.slice/crio-864c708a10eb8805f207e19953b5712a95e34a7b1a52ff58bd2fa82d2ad3f457 WatchSource:0}: Error finding container 864c708a10eb8805f207e19953b5712a95e34a7b1a52ff58bd2fa82d2ad3f457: Status 404 returned error can't find the container with id 864c708a10eb8805f207e19953b5712a95e34a7b1a52ff58bd2fa82d2ad3f457
Sep 09 07:04:37.228582 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:37.228545    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:37.228684 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:37.228661    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:37.228747 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:37.228708    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:38.112178 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.112138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" event={"ID":"f4a57d65-72e8-4306-9105-228095c56581","Type":"ContainerStarted","Data":"baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17"}
Sep 09 07:04:38.112653 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.112197    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" event={"ID":"f4a57d65-72e8-4306-9105-228095c56581","Type":"ContainerStarted","Data":"864c708a10eb8805f207e19953b5712a95e34a7b1a52ff58bd2fa82d2ad3f457"}
Sep 09 07:04:38.113719 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.113693    2562 generic.go:356] "Generic (PLEG): container finished" podID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerID="9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312" exitCode=0
Sep 09 07:04:38.113819 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.113760    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" event={"ID":"0d8c2d4f-3de5-4d05-94f4-48902a2e1477","Type":"ContainerDied","Data":"9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312"}
Sep 09 07:04:38.114136 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.114096    2562 scope.go:117] "RemoveContainer" containerID="9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312"
Sep 09 07:04:38.115133 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.115090    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:38.115225 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.115200    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:38.115291 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.115259    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:38.135933 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.135873    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" podStartSLOduration=2.135855886 podStartE2EDuration="2.135855886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:04:38.132574656 +0000 UTC m=+801.352815771" watchObservedRunningTime="2026-09-09 07:04:38.135855886 +0000 UTC m=+801.356097000"
Sep 09 07:04:38.916235 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.916187    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:38.921285 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:38.921257    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:39.123168 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:39.123134    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" event={"ID":"0d8c2d4f-3de5-4d05-94f4-48902a2e1477","Type":"ContainerStarted","Data":"d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be"}
Sep 09 07:04:39.123542 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:39.123398    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:39.124326 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:39.124305    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:04:39.917122 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:39.917081    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:39.917308 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:39.917195    2562 patch_prober.go:28] interesting pod/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 07:04:39.917308 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:39.917233    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 07:04:40.917910 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:40.917858    2562 patch_prober.go:28] interesting pod/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 07:04:40.918406 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:40.917928    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 07:04:41.917184 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:41.917153    2562 patch_prober.go:28] interesting pod/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 07:04:41.917337 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:41.917218    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 07:04:42.917841 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:42.917801    2562 patch_prober.go:28] interesting pod/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 07:04:42.918272 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:42.917869    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 07:04:43.908631 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:43.908597    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"]
Sep 09 07:04:43.910825 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:43.910805    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"
Sep 09 07:04:43.913488 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:43.913466    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:04:43.913488 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:43.913473    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:04:43.917737 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:43.917700    2562 patch_prober.go:28] interesting pod/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 07:04:43.917852 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:43.917751    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 07:04:43.920578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:43.920550    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"]
Sep 09 07:04:44.021200 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:44.021161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bf25b4-e576-4552-915f-3d117fa744cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef07c382-6xddd\" (UID: \"09bf25b4-e576-4552-915f-3d117fa744cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"
Sep 09 07:04:44.122016 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:44.121978    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bf25b4-e576-4552-915f-3d117fa744cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef07c382-6xddd\" (UID: \"09bf25b4-e576-4552-915f-3d117fa744cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"
Sep 09 07:04:44.122937 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:44.122910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bf25b4-e576-4552-915f-3d117fa744cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef07c382-6xddd\" (UID: \"09bf25b4-e576-4552-915f-3d117fa744cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"
Sep 09 07:04:44.223239 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:44.223203    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"
Sep 09 07:04:44.508727 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:44.508691    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"]
Sep 09 07:04:44.509788 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:04:44.509761    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bf25b4_e576_4552_915f_3d117fa744cf.slice/crio-6f8521f3db8a56eb42e2d99954de81cf7255f6bc499c9c33c9033fa5436ab6b6 WatchSource:0}: Error finding container 6f8521f3db8a56eb42e2d99954de81cf7255f6bc499c9c33c9033fa5436ab6b6: Status 404 returned error can't find the container with id 6f8521f3db8a56eb42e2d99954de81cf7255f6bc499c9c33c9033fa5436ab6b6
Sep 09 07:04:44.921964 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:44.921931    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:44.922470 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:44.922356    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:44.923141 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:44.923123    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:04:45.160969 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:45.160938    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd" event={"ID":"09bf25b4-e576-4552-915f-3d117fa744cf","Type":"ContainerStarted","Data":"6f8521f3db8a56eb42e2d99954de81cf7255f6bc499c9c33c9033fa5436ab6b6"}
Sep 09 07:04:45.162621 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:45.162598    2562 generic.go:356] "Generic (PLEG): container finished" podID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerID="ea1252d13363a1a9b0fc1511507e654d45ea9053616925c4be8175aa88aac5e4" exitCode=0
Sep 09 07:04:45.162716 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:45.162669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" event={"ID":"c6c0aefc-35de-4f18-a689-09cd57f6a91e","Type":"ContainerDied","Data":"ea1252d13363a1a9b0fc1511507e654d45ea9053616925c4be8175aa88aac5e4"}
Sep 09 07:04:45.163153 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:45.163129    2562 scope.go:117] "RemoveContainer" containerID="ea1252d13363a1a9b0fc1511507e654d45ea9053616925c4be8175aa88aac5e4"
Sep 09 07:04:45.164229 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:45.164194    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:45.164324 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:45.164275    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:45.164369 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:45.164324    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:46.195347 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:46.195305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" event={"ID":"c6c0aefc-35de-4f18-a689-09cd57f6a91e","Type":"ContainerStarted","Data":"9f3c809fac9fe93f49c384c1ea7cbdaddf814cb3e13147a24d7832b6814f7365"}
Sep 09 07:04:46.357286 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:46.357250    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:46.357497 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:46.357469    2562 patch_prober.go:28] interesting pod/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 07:04:46.357568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:46.357541    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 07:04:47.357431 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:47.357389    2562 patch_prober.go:28] interesting pod/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 07:04:47.357832 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:47.357467    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 07:04:48.208602 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:48.208571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd" event={"ID":"09bf25b4-e576-4552-915f-3d117fa744cf","Type":"ContainerStarted","Data":"b51b82fd900609766a3bb667bc8df601cdf5d90c680172ab6bbd88bbf6516078"}
Sep 09 07:04:48.227444 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:48.227399    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd" podStartSLOduration=1.810246542 podStartE2EDuration="5.227383796s" totalImagesPullingTime="3.417137254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:04:48.225622213 +0000 UTC m=+811.445863326" watchObservedRunningTime="2026-09-09 07:04:48.227383796 +0000 UTC m=+811.447624909"
Sep 09 07:04:48.357824 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:48.357785    2562 patch_prober.go:28] interesting pod/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 07:04:48.358222 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:48.357854    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 07:04:48.473988 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:48.473869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:04:49.215130 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:49.215079    2562 generic.go:356] "Generic (PLEG): container finished" podID="09bf25b4-e576-4552-915f-3d117fa744cf" containerID="b51b82fd900609766a3bb667bc8df601cdf5d90c680172ab6bbd88bbf6516078" exitCode=0
Sep 09 07:04:49.215295 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:49.215134    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd" event={"ID":"09bf25b4-e576-4552-915f-3d117fa744cf","Type":"ContainerDied","Data":"b51b82fd900609766a3bb667bc8df601cdf5d90c680172ab6bbd88bbf6516078"}
Sep 09 07:04:49.357937 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:49.357902    2562 patch_prober.go:28] interesting pod/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 07:04:49.358322 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:49.357952    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 07:04:50.359234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:50.359201    2562 patch_prober.go:28] interesting pod/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 09 07:04:50.359684 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:50.359262    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 09 07:04:50.422836 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:50.422804    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"
Sep 09 07:04:50.496157 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:50.495473    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bf25b4-e576-4552-915f-3d117fa744cf-maas-api-service-ca\") pod \"09bf25b4-e576-4552-915f-3d117fa744cf\" (UID: \"09bf25b4-e576-4552-915f-3d117fa744cf\") "
Sep 09 07:04:50.496518 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:50.496475    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bf25b4-e576-4552-915f-3d117fa744cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09bf25b4-e576-4552-915f-3d117fa744cf" (UID: "09bf25b4-e576-4552-915f-3d117fa744cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:04:50.596613 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:50.596580    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bf25b4-e576-4552-915f-3d117fa744cf-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:04:51.228955 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:51.228928    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"
Sep 09 07:04:51.230665 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:51.230636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd" event={"ID":"09bf25b4-e576-4552-915f-3d117fa744cf","Type":"ContainerDied","Data":"6f8521f3db8a56eb42e2d99954de81cf7255f6bc499c9c33c9033fa5436ab6b6"}
Sep 09 07:04:51.230809 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:51.230680    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f8521f3db8a56eb42e2d99954de81cf7255f6bc499c9c33c9033fa5436ab6b6"
Sep 09 07:04:51.398271 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:51.398238    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:51.398727 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:51.398623    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:51.400140 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:51.400097    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:04:56.208933 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.208889    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"]
Sep 09 07:04:56.209658 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.209288    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09bf25b4-e576-4552-915f-3d117fa744cf" containerName="revoke-keys"
Sep 09 07:04:56.209658 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.209300    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="09bf25b4-e576-4552-915f-3d117fa744cf" containerName="revoke-keys"
Sep 09 07:04:56.209658 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.209372    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="09bf25b4-e576-4552-915f-3d117fa744cf" containerName="revoke-keys"
Sep 09 07:04:56.211515 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.211494    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.213999 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.213976    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-8a9acb1e-kserve-self-signed-certs\""
Sep 09 07:04:56.221249 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.221224    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"]
Sep 09 07:04:56.352285 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.352246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-dshm\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.352285 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.352289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ddac6af-6e86-44d8-be60-507586dd9291-tls-certs\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.352561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.352337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-home\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.352561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.352354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-model-cache\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.352561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.352380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2g2b\" (UniqueName: \"kubernetes.io/projected/9ddac6af-6e86-44d8-be60-507586dd9291-kube-api-access-b2g2b\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.352561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.352428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-tmp-dir\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454010 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.453969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-dshm\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454010 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.454008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ddac6af-6e86-44d8-be60-507586dd9291-tls-certs\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454322 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.454054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-home\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454322 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.454076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-model-cache\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454322 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.454099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2g2b\" (UniqueName: \"kubernetes.io/projected/9ddac6af-6e86-44d8-be60-507586dd9291-kube-api-access-b2g2b\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454322 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.454179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-tmp-dir\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454571 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.454544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-model-cache\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454657 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.454590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-home\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.454657 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.454617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-tmp-dir\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.456469 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.456438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-dshm\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.456901 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.456879    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ddac6af-6e86-44d8-be60-507586dd9291-tls-certs\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.464677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.464606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2g2b\" (UniqueName: \"kubernetes.io/projected/9ddac6af-6e86-44d8-be60-507586dd9291-kube-api-access-b2g2b\") pod \"e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") " pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.522237 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.522203    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:04:56.803803 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:56.803773    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"]
Sep 09 07:04:56.805312 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:04:56.805283    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ddac6af_6e86_44d8_be60_507586dd9291.slice/crio-941056fdae5cfc34bc74e12ab76fbe470f79497448e529dc77ea142417ccb28a WatchSource:0}: Error finding container 941056fdae5cfc34bc74e12ab76fbe470f79497448e529dc77ea142417ccb28a: Status 404 returned error can't find the container with id 941056fdae5cfc34bc74e12ab76fbe470f79497448e529dc77ea142417ccb28a
Sep 09 07:04:57.262627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:57.262591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52" event={"ID":"9ddac6af-6e86-44d8-be60-507586dd9291","Type":"ContainerStarted","Data":"cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0"}
Sep 09 07:04:57.263087 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:57.262638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52" event={"ID":"9ddac6af-6e86-44d8-be60-507586dd9291","Type":"ContainerStarted","Data":"941056fdae5cfc34bc74e12ab76fbe470f79497448e529dc77ea142417ccb28a"}
Sep 09 07:04:57.285691 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:57.285641    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52" podStartSLOduration=1.285624512 podStartE2EDuration="1.285624512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:04:57.282499055 +0000 UTC m=+820.502740171" watchObservedRunningTime="2026-09-09 07:04:57.285624512 +0000 UTC m=+820.505865626"
Sep 09 07:04:59.283374 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:59.283338    2562 generic.go:356] "Generic (PLEG): container finished" podID="f4a57d65-72e8-4306-9105-228095c56581" containerID="baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17" exitCode=0
Sep 09 07:04:59.283802 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:59.283392    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" event={"ID":"f4a57d65-72e8-4306-9105-228095c56581","Type":"ContainerDied","Data":"baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17"}
Sep 09 07:04:59.283802 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:59.283752    2562 scope.go:117] "RemoveContainer" containerID="baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17"
Sep 09 07:04:59.284768 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:59.284732    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:59.284886 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:59.284824    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:04:59.284886 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:04:59.284872    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:05:00.294622 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:00.294590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" event={"ID":"f4a57d65-72e8-4306-9105-228095c56581","Type":"ContainerStarted","Data":"3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737"}
Sep 09 07:05:00.680323 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:00.680278    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"]
Sep 09 07:05:00.690599 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:00.690564    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef07c382-6xddd"]
Sep 09 07:05:00.814736 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:00.814699    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"]
Sep 09 07:05:00.814974 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:00.814950    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy" containerID="cri-o://d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be" gracePeriod=30
Sep 09 07:05:00.915772 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:00.915730    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:05:00.915974 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:00.915855    2562 patch_prober.go:28] interesting pod/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 07:05:00.916056 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:00.915999    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 07:05:01.229995 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:01.229952    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09bf25b4-e576-4552-915f-3d117fa744cf" path="/var/lib/kubelet/pods/09bf25b4-e576-4552-915f-3d117fa744cf/volumes"
Sep 09 07:05:01.823482 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:01.823449    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"]
Sep 09 07:05:01.833030 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:01.833000    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:01.836620 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:01.836597    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-83162d4f-gw-openshift-default-dockercfg-5z288\""
Sep 09 07:05:01.845706 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:01.845669    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"]
Sep 09 07:05:01.917540 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:01.917502    2562 patch_prober.go:28] interesting pod/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 07:05:01.917834 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:01.917806    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 07:05:02.012117 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e971da7-f465-4a01-a825-d9456181a296-istio-podinfo\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.012339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-credential-socket\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.012339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgdhj\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-kube-api-access-dgdhj\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.012339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e971da7-f465-4a01-a825-d9456181a296-istiod-ca-cert\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.012339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-certs\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.012558 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-istio-token\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.012558 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-envoy\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.012558 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-data\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.012558 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.012431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-socket\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113004 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.112901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-socket\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113004 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.112962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e971da7-f465-4a01-a825-d9456181a296-istio-podinfo\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113004 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.112997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-credential-socket\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgdhj\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-kube-api-access-dgdhj\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e971da7-f465-4a01-a825-d9456181a296-istiod-ca-cert\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-certs\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-istio-token\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-envoy\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-data\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113712 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-socket\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113712 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113552    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-data\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113852 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113828    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-certs\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113913 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113896    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-credential-socket\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.113970 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.113936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e971da7-f465-4a01-a825-d9456181a296-istiod-ca-cert\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.115491 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.115464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-envoy\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.115772 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.115755    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e971da7-f465-4a01-a825-d9456181a296-istio-podinfo\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.122290 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.122268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-istio-token\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.122474 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.122456    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgdhj\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-kube-api-access-dgdhj\") pod \"e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") " pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.144348 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.144308    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:02.453621 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.453593    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"]
Sep 09 07:05:02.455801 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:05:02.455759    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e971da7_f465_4a01_a825_d9456181a296.slice/crio-3bbd781844306eb13f8a74a13e5d1641b249a5432cd4ddcd9e0299d22eba1842 WatchSource:0}: Error finding container 3bbd781844306eb13f8a74a13e5d1641b249a5432cd4ddcd9e0299d22eba1842: Status 404 returned error can't find the container with id 3bbd781844306eb13f8a74a13e5d1641b249a5432cd4ddcd9e0299d22eba1842
Sep 09 07:05:02.458263 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.458223    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:05:02.458347 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.458308    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:05:02.458395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.458353    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:05:02.916396 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.916362    2562 patch_prober.go:28] interesting pod/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 07:05:02.916792 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:02.916418    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 07:05:03.313091 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:03.312992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" event={"ID":"5e971da7-f465-4a01-a825-d9456181a296","Type":"ContainerStarted","Data":"e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0"}
Sep 09 07:05:03.313091 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:03.313044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" event={"ID":"5e971da7-f465-4a01-a825-d9456181a296","Type":"ContainerStarted","Data":"3bbd781844306eb13f8a74a13e5d1641b249a5432cd4ddcd9e0299d22eba1842"}
Sep 09 07:05:03.344401 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:03.344339    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podStartSLOduration=2.344315006 podStartE2EDuration="2.344315006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:05:03.340318306 +0000 UTC m=+826.560559415" watchObservedRunningTime="2026-09-09 07:05:03.344315006 +0000 UTC m=+826.564556119"
Sep 09 07:05:03.916214 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:03.916177    2562 patch_prober.go:28] interesting pod/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 07:05:03.916389 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:03.916247    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 07:05:04.144908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:04.144868    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:04.149996 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:04.149969    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:04.318688 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:04.318602    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:04.320019 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:04.319996    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:04.916080 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:04.916041    2562 patch_prober.go:28] interesting pod/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 07:05:04.916287 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:04.916131    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 07:05:05.920132 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:05.920064    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:05:05.920604 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:05.920582    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:05:05.921360 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:05.921343    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:05:06.084446 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.084383    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:05:06.156778 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.156742    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-envoy\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.156976 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.156832    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-podinfo\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.156976 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.156860    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-token\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.156976 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.156887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-data\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.156976 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.156921    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-socket\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.156976 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.156960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxmj6\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-kube-api-access-pxmj6\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.157275 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157008    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-certs\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.157275 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157037    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-credential-socket\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.157275 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157068    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istiod-ca-cert\") pod \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\" (UID: \"0d8c2d4f-3de5-4d05-94f4-48902a2e1477\") "
Sep 09 07:05:06.157440 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157267    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:06.157505 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157450    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.157687 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157627    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:06.157687 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157671    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:06.157845 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157805    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:06.157892 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.157855    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:05:06.159591 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.159563    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:05:06.159682 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.159592    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:05:06.159682 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.159631    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:06.159682 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.159665    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-kube-api-access-pxmj6" (OuterVolumeSpecName: "kube-api-access-pxmj6") pod "0d8c2d4f-3de5-4d05-94f4-48902a2e1477" (UID: "0d8c2d4f-3de5-4d05-94f4-48902a2e1477"). InnerVolumeSpecName "kube-api-access-pxmj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:05:06.258395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.258364    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.258395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.258391    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.258395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.258402    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.258649 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.258410    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.258649 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.258419    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxmj6\" (UniqueName: \"kubernetes.io/projected/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-kube-api-access-pxmj6\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.258649 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.258429    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.258649 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.258438    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.258649 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.258445    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d8c2d4f-3de5-4d05-94f4-48902a2e1477-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:06.330541 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.330499    2562 generic.go:356] "Generic (PLEG): container finished" podID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerID="d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be" exitCode=0
Sep 09 07:05:06.330725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.330568    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"
Sep 09 07:05:06.330725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.330634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" event={"ID":"0d8c2d4f-3de5-4d05-94f4-48902a2e1477","Type":"ContainerDied","Data":"d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be"}
Sep 09 07:05:06.330725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.330687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr" event={"ID":"0d8c2d4f-3de5-4d05-94f4-48902a2e1477","Type":"ContainerDied","Data":"24dc41b8fd641731e915cec6583cf24c4c0af1bea0449c1164e790948c244a8b"}
Sep 09 07:05:06.330725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.330710    2562 scope.go:117] "RemoveContainer" containerID="d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be"
Sep 09 07:05:06.340587 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.340566    2562 scope.go:117] "RemoveContainer" containerID="9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312"
Sep 09 07:05:06.348504 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.348487    2562 scope.go:117] "RemoveContainer" containerID="d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be"
Sep 09 07:05:06.348748 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:05:06.348731    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be\": container with ID starting with d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be not found: ID does not exist" containerID="d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be"
Sep 09 07:05:06.348788 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.348757    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be"} err="failed to get container status \"d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be\": rpc error: code = NotFound desc = could not find container \"d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be\": container with ID starting with d8bab67818e296e0b1ccff15e24019668bb34e2c72aac0e9567fb2814646b2be not found: ID does not exist"
Sep 09 07:05:06.348788 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.348775    2562 scope.go:117] "RemoveContainer" containerID="9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312"
Sep 09 07:05:06.349022 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:05:06.348994    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312\": container with ID starting with 9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312 not found: ID does not exist" containerID="9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312"
Sep 09 07:05:06.349060 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.349029    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312"} err="failed to get container status \"9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312\": rpc error: code = NotFound desc = could not find container \"9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312\": container with ID starting with 9d73f5b7575d6104da3e86a95fe2c46178dacde1d23512ffbd13da93ac73e312 not found: ID does not exist"
Sep 09 07:05:06.356568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.356545    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"]
Sep 09 07:05:06.361685 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.361665    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ef07c382-openshift-default-59465b7fd4-qrggr"]
Sep 09 07:05:06.522818 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.522779    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:05:06.522994 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.522831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:05:06.537544 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:06.537486    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:05:07.231066 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:07.231033    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" path="/var/lib/kubelet/pods/0d8c2d4f-3de5-4d05-94f4-48902a2e1477/volumes"
Sep 09 07:05:07.348672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:07.348644    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:05:09.674028 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:09.673990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:05:19.550407 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:19.549815    2562 patch_prober.go:28] interesting pod/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": read tcp 10.134.0.2:56792->10.134.0.45:15021: read: connection reset by peer" start-of-body=
Sep 09 07:05:19.550407 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:19.549879    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": read tcp 10.134.0.2:56792->10.134.0.45:15021: read: connection reset by peer"
Sep 09 07:05:20.429967 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:20.429298    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e971da7-f465-4a01-a825-d9456181a296" containerID="e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0" exitCode=0
Sep 09 07:05:20.429967 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:20.429394    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" event={"ID":"5e971da7-f465-4a01-a825-d9456181a296","Type":"ContainerDied","Data":"e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0"}
Sep 09 07:05:20.430268 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:20.430005    2562 scope.go:117] "RemoveContainer" containerID="e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0"
Sep 09 07:05:20.431485 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:20.431454    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:05:20.431572 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:20.431526    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:05:20.431572 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:20.431561    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:05:21.437823 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:21.437790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" event={"ID":"5e971da7-f465-4a01-a825-d9456181a296","Type":"ContainerStarted","Data":"efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea"}
Sep 09 07:05:22.144823 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:22.144780    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:22.145017 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:22.144994    2562 patch_prober.go:28] interesting pod/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 07:05:22.145079 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:22.145047    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 07:05:23.144951 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:23.144912    2562 patch_prober.go:28] interesting pod/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 07:05:23.145378 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:23.144969    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 07:05:24.145302 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:24.145259    2562 patch_prober.go:28] interesting pod/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 07:05:24.145676 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:24.145322    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 07:05:25.145448 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:25.145413    2562 patch_prober.go:28] interesting pod/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 07:05:25.145835 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:25.145470    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 07:05:26.144856 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:26.144817    2562 patch_prober.go:28] interesting pod/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 07:05:26.145088 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:26.144895    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 07:05:26.396164 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:26.396055    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:05:27.148604 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:27.148570    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:27.148936 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:27.148912    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:27.149584 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:27.149566    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:29.814473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.814437    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"]
Sep 09 07:05:29.814956 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.814777    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy"
Sep 09 07:05:29.814956 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.814787    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy"
Sep 09 07:05:29.814956 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.814856    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy"
Sep 09 07:05:29.814956 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.814867    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy"
Sep 09 07:05:29.817905 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.817887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"
Sep 09 07:05:29.820803 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.820780    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:05:29.820909 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.820783    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:05:29.833576 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.833555    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"]
Sep 09 07:05:29.884198 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.884170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd484f6-afae-4904-9ee3-57b557e26858-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj\" (UID: \"0dd484f6-afae-4904-9ee3-57b557e26858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"
Sep 09 07:05:29.985345 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.985312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd484f6-afae-4904-9ee3-57b557e26858-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj\" (UID: \"0dd484f6-afae-4904-9ee3-57b557e26858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"
Sep 09 07:05:29.985924 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:29.985904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd484f6-afae-4904-9ee3-57b557e26858-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj\" (UID: \"0dd484f6-afae-4904-9ee3-57b557e26858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"
Sep 09 07:05:30.127677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:30.127587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"
Sep 09 07:05:30.403662 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:30.403630    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"]
Sep 09 07:05:30.404897 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:05:30.404872    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd484f6_afae_4904_9ee3_57b557e26858.slice/crio-5745f768c82de0abfdd9b64de8e21abd1e51895b41aaaf43d2577c24b0ffec8a WatchSource:0}: Error finding container 5745f768c82de0abfdd9b64de8e21abd1e51895b41aaaf43d2577c24b0ffec8a: Status 404 returned error can't find the container with id 5745f768c82de0abfdd9b64de8e21abd1e51895b41aaaf43d2577c24b0ffec8a
Sep 09 07:05:30.493341 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:30.493310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj" event={"ID":"0dd484f6-afae-4904-9ee3-57b557e26858","Type":"ContainerStarted","Data":"5745f768c82de0abfdd9b64de8e21abd1e51895b41aaaf43d2577c24b0ffec8a"}
Sep 09 07:05:31.500566 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:31.500526    2562 generic.go:356] "Generic (PLEG): container finished" podID="0dd484f6-afae-4904-9ee3-57b557e26858" containerID="5399232b9e4ab0e3a3dc86f5e19cdb58c87f1376625f4fdc3d823036171d212b" exitCode=0
Sep 09 07:05:31.500964 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:31.500584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj" event={"ID":"0dd484f6-afae-4904-9ee3-57b557e26858","Type":"ContainerDied","Data":"5399232b9e4ab0e3a3dc86f5e19cdb58c87f1376625f4fdc3d823036171d212b"}
Sep 09 07:05:32.637436 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:32.637404    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"
Sep 09 07:05:32.711737 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:32.711707    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd484f6-afae-4904-9ee3-57b557e26858-maas-api-service-ca\") pod \"0dd484f6-afae-4904-9ee3-57b557e26858\" (UID: \"0dd484f6-afae-4904-9ee3-57b557e26858\") "
Sep 09 07:05:32.712014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:32.711993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd484f6-afae-4904-9ee3-57b557e26858-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dd484f6-afae-4904-9ee3-57b557e26858" (UID: "0dd484f6-afae-4904-9ee3-57b557e26858"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:05:32.812656 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:32.812585    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd484f6-afae-4904-9ee3-57b557e26858-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:33.513365 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:33.513283    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"
Sep 09 07:05:33.513509 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:33.513293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj" event={"ID":"0dd484f6-afae-4904-9ee3-57b557e26858","Type":"ContainerDied","Data":"5745f768c82de0abfdd9b64de8e21abd1e51895b41aaaf43d2577c24b0ffec8a"}
Sep 09 07:05:33.513509 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:33.513420    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5745f768c82de0abfdd9b64de8e21abd1e51895b41aaaf43d2577c24b0ffec8a"
Sep 09 07:05:34.641474 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.641430    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"]
Sep 09 07:05:34.641842 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.641800    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd484f6-afae-4904-9ee3-57b557e26858" containerName="revoke-keys"
Sep 09 07:05:34.641842 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.641812    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd484f6-afae-4904-9ee3-57b557e26858" containerName="revoke-keys"
Sep 09 07:05:34.641842 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.641821    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy"
Sep 09 07:05:34.641842 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.641829    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8c2d4f-3de5-4d05-94f4-48902a2e1477" containerName="istio-proxy"
Sep 09 07:05:34.641993 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.641882    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd484f6-afae-4904-9ee3-57b557e26858" containerName="revoke-keys"
Sep 09 07:05:34.644916 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.644899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.648661 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.648637    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-40c98b\"/\"default-dockercfg-87mm6\""
Sep 09 07:05:34.648794 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.648738    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-40c98b\"/\"openshift-service-ca.crt\""
Sep 09 07:05:34.648862 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.648795    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-40c98b\"/\"kube-root-ca.crt\""
Sep 09 07:05:34.648919 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.648872    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-40c98b\"/\"autores-model-40c98b-kserve-self-signed-certs\""
Sep 09 07:05:34.655556 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.655533    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"]
Sep 09 07:05:34.729395 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.729359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tls-certs\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.729561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.729479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-home\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.729561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.729521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tmp-dir\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.729561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.729547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-model-cache\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.729673 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.729571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-dshm\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.729673 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.729594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfcr6\" (UniqueName: \"kubernetes.io/projected/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-kube-api-access-cfcr6\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.830791 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.830756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-home\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.830983 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.830801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tmp-dir\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.830983 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.830824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-model-cache\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.830983 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.830840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-dshm\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.830983 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.830965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfcr6\" (UniqueName: \"kubernetes.io/projected/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-kube-api-access-cfcr6\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.831235 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.831041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tls-certs\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.831235 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.831215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-home\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.831326 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.831252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-model-cache\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.831326 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.831282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tmp-dir\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.833144 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.833091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-dshm\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.833577 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.833558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tls-certs\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.840471 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.840449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfcr6\" (UniqueName: \"kubernetes.io/projected/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-kube-api-access-cfcr6\") pod \"autores-model-40c98b-kserve-55997f5fc9-z6xcd\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") " pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:34.954868 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:34.954841    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:35.238001 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:35.237977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"]
Sep 09 07:05:35.239780 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:05:35.239747    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cebb5aa_7615_428c_b34b_00cfcb2b27b0.slice/crio-962d1acded1a7a8cdaa4e5e1be1a0b336061f32b7f59fea0a215a5961ee76d27 WatchSource:0}: Error finding container 962d1acded1a7a8cdaa4e5e1be1a0b336061f32b7f59fea0a215a5961ee76d27: Status 404 returned error can't find the container with id 962d1acded1a7a8cdaa4e5e1be1a0b336061f32b7f59fea0a215a5961ee76d27
Sep 09 07:05:35.526592 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:35.526505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd" event={"ID":"6cebb5aa-7615-428c-b34b-00cfcb2b27b0","Type":"ContainerStarted","Data":"9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b"}
Sep 09 07:05:35.526592 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:35.526551    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd" event={"ID":"6cebb5aa-7615-428c-b34b-00cfcb2b27b0","Type":"ContainerStarted","Data":"962d1acded1a7a8cdaa4e5e1be1a0b336061f32b7f59fea0a215a5961ee76d27"}
Sep 09 07:05:35.547971 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:35.547921    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd" podStartSLOduration=1.5479068310000001 podStartE2EDuration="1.547906831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:05:35.545479216 +0000 UTC m=+858.765720331" watchObservedRunningTime="2026-09-09 07:05:35.547906831 +0000 UTC m=+858.768147943"
Sep 09 07:05:36.693505 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:36.693470    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"]
Sep 09 07:05:36.697914 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:36.697883    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83162d4f-md5tj"]
Sep 09 07:05:36.934576 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:36.934533    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"]
Sep 09 07:05:36.934910 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:36.934852    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy" containerID="cri-o://efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea" gracePeriod=30
Sep 09 07:05:37.229883 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:37.229851    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd484f6-afae-4904-9ee3-57b557e26858" path="/var/lib/kubelet/pods/0dd484f6-afae-4904-9ee3-57b557e26858/volumes"
Sep 09 07:05:39.360908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:39.360865    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:05:42.180679 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.180653    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:42.304767 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.304686    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-socket\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.304767 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.304722    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-envoy\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.304767 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.304762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-credential-socket\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.305041 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.304784    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e971da7-f465-4a01-a825-d9456181a296-istio-podinfo\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.305041 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.304872    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e971da7-f465-4a01-a825-d9456181a296-istiod-ca-cert\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.305041 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.304907    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-certs\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.305041 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.304919    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:42.305041 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.304982    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-data\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.305041 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305049    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-istio-token\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.305534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305083    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:42.305534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305133    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgdhj\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-kube-api-access-dgdhj\") pod \"5e971da7-f465-4a01-a825-d9456181a296\" (UID: \"5e971da7-f465-4a01-a825-d9456181a296\") "
Sep 09 07:05:42.305534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305146    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:42.305534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305255    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:42.305534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305343    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e971da7-f465-4a01-a825-d9456181a296-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:05:42.305534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305522    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e971da7-f465-4a01-a825-d9456181a296-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.305802 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305543    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.305802 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305556    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.305802 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305569    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.305802 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.305583    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.307167 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.307125    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:05:42.307268 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.307196    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-kube-api-access-dgdhj" (OuterVolumeSpecName: "kube-api-access-dgdhj") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "kube-api-access-dgdhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:05:42.307401 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.307383    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:05:42.307446 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.307393    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e971da7-f465-4a01-a825-d9456181a296-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e971da7-f465-4a01-a825-d9456181a296" (UID: "5e971da7-f465-4a01-a825-d9456181a296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:05:42.406966 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.406928    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.406966 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.406961    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgdhj\" (UniqueName: \"kubernetes.io/projected/5e971da7-f465-4a01-a825-d9456181a296-kube-api-access-dgdhj\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.406966 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.406971    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e971da7-f465-4a01-a825-d9456181a296-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.407229 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.406979    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e971da7-f465-4a01-a825-d9456181a296-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:05:42.567643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.567557    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e971da7-f465-4a01-a825-d9456181a296" containerID="efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea" exitCode=0
Sep 09 07:05:42.567643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.567600    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"
Sep 09 07:05:42.567827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.567645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" event={"ID":"5e971da7-f465-4a01-a825-d9456181a296","Type":"ContainerDied","Data":"efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea"}
Sep 09 07:05:42.567827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.567709    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" event={"ID":"5e971da7-f465-4a01-a825-d9456181a296","Type":"ContainerDied","Data":"3bbd781844306eb13f8a74a13e5d1641b249a5432cd4ddcd9e0299d22eba1842"}
Sep 09 07:05:42.567827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.567740    2562 scope.go:117] "RemoveContainer" containerID="efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea"
Sep 09 07:05:42.585459 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.585435    2562 scope.go:117] "RemoveContainer" containerID="e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0"
Sep 09 07:05:42.593889 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.593860    2562 scope.go:117] "RemoveContainer" containerID="efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea"
Sep 09 07:05:42.594219 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:05:42.594182    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea\": container with ID starting with efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea not found: ID does not exist" containerID="efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea"
Sep 09 07:05:42.594303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.594230    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea"} err="failed to get container status \"efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea\": rpc error: code = NotFound desc = could not find container \"efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea\": container with ID starting with efbdd4e36a6c195d3a03274ad278bc5e06f96559b9765a3381fdab98b1f834ea not found: ID does not exist"
Sep 09 07:05:42.594303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.594260    2562 scope.go:117] "RemoveContainer" containerID="e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0"
Sep 09 07:05:42.594529 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:05:42.594508    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0\": container with ID starting with e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0 not found: ID does not exist" containerID="e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0"
Sep 09 07:05:42.594589 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.594538    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0"} err="failed to get container status \"e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0\": rpc error: code = NotFound desc = could not find container \"e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0\": container with ID starting with e57b3d742ca338ca3d0cdb5a348fa53998eb082690c3f528b8b2ad474f33dbb0 not found: ID does not exist"
Sep 09 07:05:42.598638 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.598616    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"]
Sep 09 07:05:42.605092 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:42.605072    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46"]
Sep 09 07:05:43.150645 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:43.150606    2562 patch_prober.go:28] interesting pod/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 07:05:43.150831 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:43.150679    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-83162d4f-gw-openshift-default-7c6d467656-4vz46" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 07:05:43.230077 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:43.230045    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e971da7-f465-4a01-a825-d9456181a296" path="/var/lib/kubelet/pods/5e971da7-f465-4a01-a825-d9456181a296/volumes"
Sep 09 07:05:44.955059 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:44.955024    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:44.955473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:44.955122    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:44.967837 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:44.967814    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:45.598223 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:45.598195    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:05:56.894663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:05:56.894627    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:06:06.067953 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.067921    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"]
Sep 09 07:06:06.068586 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.068463    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy"
Sep 09 07:06:06.068586 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.068483    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy"
Sep 09 07:06:06.068586 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.068495    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy"
Sep 09 07:06:06.068586 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.068505    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy"
Sep 09 07:06:06.068808 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.068588    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy"
Sep 09 07:06:06.068808 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.068601    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e971da7-f465-4a01-a825-d9456181a296" containerName="istio-proxy"
Sep 09 07:06:06.072071 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.072051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"
Sep 09 07:06:06.076614 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.076588    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:06:06.076723 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.076601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:06:06.091015 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.090990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"]
Sep 09 07:06:06.136717 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.136688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b34b39-1741-426d-b5e0-5a6eb5cb0b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-40c98b-766gk\" (UID: \"07b34b39-1741-426d-b5e0-5a6eb5cb0b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"
Sep 09 07:06:06.237993 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.237963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b34b39-1741-426d-b5e0-5a6eb5cb0b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-40c98b-766gk\" (UID: \"07b34b39-1741-426d-b5e0-5a6eb5cb0b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"
Sep 09 07:06:06.238622 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.238603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b34b39-1741-426d-b5e0-5a6eb5cb0b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-40c98b-766gk\" (UID: \"07b34b39-1741-426d-b5e0-5a6eb5cb0b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"
Sep 09 07:06:06.405234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.405157    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"
Sep 09 07:06:06.700928 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:06.700884    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"]
Sep 09 07:06:06.703018 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:06:06.702990    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b34b39_1741_426d_b5e0_5a6eb5cb0b6f.slice/crio-4b115f530f8f3bf1e948cbd83622a7430701cbe9dc79bfd039356666c11914cd WatchSource:0}: Error finding container 4b115f530f8f3bf1e948cbd83622a7430701cbe9dc79bfd039356666c11914cd: Status 404 returned error can't find the container with id 4b115f530f8f3bf1e948cbd83622a7430701cbe9dc79bfd039356666c11914cd
Sep 09 07:06:07.712643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:07.712605    2562 generic.go:356] "Generic (PLEG): container finished" podID="07b34b39-1741-426d-b5e0-5a6eb5cb0b6f" containerID="d27e8a329d02f1d9786a3ef90e9d7e9017ccf49aff1d173999d1825e21dc635b" exitCode=0
Sep 09 07:06:07.713046 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:07.712695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk" event={"ID":"07b34b39-1741-426d-b5e0-5a6eb5cb0b6f","Type":"ContainerDied","Data":"d27e8a329d02f1d9786a3ef90e9d7e9017ccf49aff1d173999d1825e21dc635b"}
Sep 09 07:06:07.713046 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:07.712786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk" event={"ID":"07b34b39-1741-426d-b5e0-5a6eb5cb0b6f","Type":"ContainerStarted","Data":"4b115f530f8f3bf1e948cbd83622a7430701cbe9dc79bfd039356666c11914cd"}
Sep 09 07:06:08.847988 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:08.847964    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"
Sep 09 07:06:08.968799 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:08.968724    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b34b39-1741-426d-b5e0-5a6eb5cb0b6f-maas-api-service-ca\") pod \"07b34b39-1741-426d-b5e0-5a6eb5cb0b6f\" (UID: \"07b34b39-1741-426d-b5e0-5a6eb5cb0b6f\") "
Sep 09 07:06:08.969081 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:08.969059    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b34b39-1741-426d-b5e0-5a6eb5cb0b6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07b34b39-1741-426d-b5e0-5a6eb5cb0b6f" (UID: "07b34b39-1741-426d-b5e0-5a6eb5cb0b6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:06:09.069614 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:09.069584    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b34b39-1741-426d-b5e0-5a6eb5cb0b6f-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:09.725315 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:09.725283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk" event={"ID":"07b34b39-1741-426d-b5e0-5a6eb5cb0b6f","Type":"ContainerDied","Data":"4b115f530f8f3bf1e948cbd83622a7430701cbe9dc79bfd039356666c11914cd"}
Sep 09 07:06:09.725493 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:09.725332    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b115f530f8f3bf1e948cbd83622a7430701cbe9dc79bfd039356666c11914cd"
Sep 09 07:06:09.725493 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:09.725350    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"
Sep 09 07:06:11.416163 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:11.416126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:06:13.628012 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.627978    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"]
Sep 09 07:06:13.628401 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.628375    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b34b39-1741-426d-b5e0-5a6eb5cb0b6f" containerName="revoke-keys"
Sep 09 07:06:13.628401 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.628387    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b34b39-1741-426d-b5e0-5a6eb5cb0b6f" containerName="revoke-keys"
Sep 09 07:06:13.628481 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.628457    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b34b39-1741-426d-b5e0-5a6eb5cb0b6f" containerName="revoke-keys"
Sep 09 07:06:13.631565 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.631550    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"
Sep 09 07:06:13.634540 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.634521    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:06:13.634672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.634628    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:06:13.641925 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.641899    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"]
Sep 09 07:06:13.713815 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.713779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76c03cf-6a4b-4995-8063-60f1706d37bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eec132b-th65p\" (UID: \"c76c03cf-6a4b-4995-8063-60f1706d37bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"
Sep 09 07:06:13.814986 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.814946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76c03cf-6a4b-4995-8063-60f1706d37bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eec132b-th65p\" (UID: \"c76c03cf-6a4b-4995-8063-60f1706d37bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"
Sep 09 07:06:13.815587 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.815567    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76c03cf-6a4b-4995-8063-60f1706d37bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7eec132b-th65p\" (UID: \"c76c03cf-6a4b-4995-8063-60f1706d37bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"
Sep 09 07:06:13.943003 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:13.942964    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"
Sep 09 07:06:14.214554 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:14.214525    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"]
Sep 09 07:06:14.215875 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:06:14.215849    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76c03cf_6a4b_4995_8063_60f1706d37bb.slice/crio-f04194e81bb65bfcc11d55ca2a0c1c091971f9b23086870c35294840c72bb538 WatchSource:0}: Error finding container f04194e81bb65bfcc11d55ca2a0c1c091971f9b23086870c35294840c72bb538: Status 404 returned error can't find the container with id f04194e81bb65bfcc11d55ca2a0c1c091971f9b23086870c35294840c72bb538
Sep 09 07:06:14.759910 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:14.759822    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p" event={"ID":"c76c03cf-6a4b-4995-8063-60f1706d37bb","Type":"ContainerStarted","Data":"ca18ba04c6b1a4da49f588ece0e51208b146b0213f6fd29badc1669b75346ca8"}
Sep 09 07:06:14.759910 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:14.759881    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p" event={"ID":"c76c03cf-6a4b-4995-8063-60f1706d37bb","Type":"ContainerStarted","Data":"f04194e81bb65bfcc11d55ca2a0c1c091971f9b23086870c35294840c72bb538"}
Sep 09 07:06:14.782582 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:14.782534    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p" podStartSLOduration=1.7825208099999998 podStartE2EDuration="1.78252081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:06:14.780793739 +0000 UTC m=+898.001034853" watchObservedRunningTime="2026-09-09 07:06:14.78252081 +0000 UTC m=+898.002761922"
Sep 09 07:06:15.766962 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:15.766927    2562 generic.go:356] "Generic (PLEG): container finished" podID="c76c03cf-6a4b-4995-8063-60f1706d37bb" containerID="ca18ba04c6b1a4da49f588ece0e51208b146b0213f6fd29badc1669b75346ca8" exitCode=0
Sep 09 07:06:15.767371 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:15.767017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p" event={"ID":"c76c03cf-6a4b-4995-8063-60f1706d37bb","Type":"ContainerDied","Data":"ca18ba04c6b1a4da49f588ece0e51208b146b0213f6fd29badc1669b75346ca8"}
Sep 09 07:06:16.925303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:16.925277    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"
Sep 09 07:06:17.044865 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.044769    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"]
Sep 09 07:06:17.048325 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.048299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76c03cf-6a4b-4995-8063-60f1706d37bb-maas-api-service-ca\") pod \"c76c03cf-6a4b-4995-8063-60f1706d37bb\" (UID: \"c76c03cf-6a4b-4995-8063-60f1706d37bb\") "
Sep 09 07:06:17.048734 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.048694    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76c03cf-6a4b-4995-8063-60f1706d37bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c76c03cf-6a4b-4995-8063-60f1706d37bb" (UID: "c76c03cf-6a4b-4995-8063-60f1706d37bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:06:17.051485 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.051462    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-40c98b-766gk"]
Sep 09 07:06:17.150045 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.149995    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76c03cf-6a4b-4995-8063-60f1706d37bb-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:17.193237 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.193205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:06:17.195556 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.195528    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:06:17.229998 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.229971    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b34b39-1741-426d-b5e0-5a6eb5cb0b6f" path="/var/lib/kubelet/pods/07b34b39-1741-426d-b5e0-5a6eb5cb0b6f/volumes"
Sep 09 07:06:17.781028 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.781000    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"
Sep 09 07:06:17.781190 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.781027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p" event={"ID":"c76c03cf-6a4b-4995-8063-60f1706d37bb","Type":"ContainerDied","Data":"f04194e81bb65bfcc11d55ca2a0c1c091971f9b23086870c35294840c72bb538"}
Sep 09 07:06:17.781190 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:17.781075    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f04194e81bb65bfcc11d55ca2a0c1c091971f9b23086870c35294840c72bb538"
Sep 09 07:06:20.793609 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:20.793565    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"]
Sep 09 07:06:22.614286 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:22.614255    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"]
Sep 09 07:06:24.687453 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:24.687418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:06:25.735926 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:25.735888    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"]
Sep 09 07:06:25.746606 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:25.746580    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7eec132b-th65p"]
Sep 09 07:06:27.230063 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:27.230028    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76c03cf-6a4b-4995-8063-60f1706d37bb" path="/var/lib/kubelet/pods/c76c03cf-6a4b-4995-8063-60f1706d37bb/volumes"
Sep 09 07:06:35.812742 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:35.812700    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd" podUID="6cebb5aa-7615-428c-b34b-00cfcb2b27b0" containerName="main" containerID="cri-o://9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b" gracePeriod=45
Sep 09 07:06:36.069958 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.069896    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:06:36.128935 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.128903    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-dshm\") pod \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") "
Sep 09 07:06:36.129131 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.128990    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tmp-dir\") pod \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") "
Sep 09 07:06:36.129131 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129012    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfcr6\" (UniqueName: \"kubernetes.io/projected/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-kube-api-access-cfcr6\") pod \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") "
Sep 09 07:06:36.129131 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129041    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-home\") pod \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") "
Sep 09 07:06:36.129131 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129056    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-model-cache\") pod \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") "
Sep 09 07:06:36.129394 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129115    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tls-certs\") pod \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\" (UID: \"6cebb5aa-7615-428c-b34b-00cfcb2b27b0\") "
Sep 09 07:06:36.129495 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129348    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cebb5aa-7615-428c-b34b-00cfcb2b27b0" (UID: "6cebb5aa-7615-428c-b34b-00cfcb2b27b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:06:36.129554 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129339    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cebb5aa-7615-428c-b34b-00cfcb2b27b0" (UID: "6cebb5aa-7615-428c-b34b-00cfcb2b27b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:06:36.129554 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129364    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-home" (OuterVolumeSpecName: "home") pod "6cebb5aa-7615-428c-b34b-00cfcb2b27b0" (UID: "6cebb5aa-7615-428c-b34b-00cfcb2b27b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:06:36.129741 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129717    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:36.129741 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129739    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:36.129932 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.129748    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:36.131280 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.131260    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-dshm" (OuterVolumeSpecName: "dshm") pod "6cebb5aa-7615-428c-b34b-00cfcb2b27b0" (UID: "6cebb5aa-7615-428c-b34b-00cfcb2b27b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:06:36.131527 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.131500    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-kube-api-access-cfcr6" (OuterVolumeSpecName: "kube-api-access-cfcr6") pod "6cebb5aa-7615-428c-b34b-00cfcb2b27b0" (UID: "6cebb5aa-7615-428c-b34b-00cfcb2b27b0"). InnerVolumeSpecName "kube-api-access-cfcr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:06:36.131611 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.131527    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cebb5aa-7615-428c-b34b-00cfcb2b27b0" (UID: "6cebb5aa-7615-428c-b34b-00cfcb2b27b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:06:36.230693 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.230659    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:36.230693 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.230688    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfcr6\" (UniqueName: \"kubernetes.io/projected/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-kube-api-access-cfcr6\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:36.230693 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.230699    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cebb5aa-7615-428c-b34b-00cfcb2b27b0-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:36.892348 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.892314    2562 generic.go:356] "Generic (PLEG): container finished" podID="6cebb5aa-7615-428c-b34b-00cfcb2b27b0" containerID="9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b" exitCode=0
Sep 09 07:06:36.892834 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.892364    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"
Sep 09 07:06:36.892834 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.892366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd" event={"ID":"6cebb5aa-7615-428c-b34b-00cfcb2b27b0","Type":"ContainerDied","Data":"9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b"}
Sep 09 07:06:36.892834 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.892485    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd" event={"ID":"6cebb5aa-7615-428c-b34b-00cfcb2b27b0","Type":"ContainerDied","Data":"962d1acded1a7a8cdaa4e5e1be1a0b336061f32b7f59fea0a215a5961ee76d27"}
Sep 09 07:06:36.892834 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.892509    2562 scope.go:117] "RemoveContainer" containerID="9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b"
Sep 09 07:06:36.902017 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.901983    2562 scope.go:117] "RemoveContainer" containerID="9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b"
Sep 09 07:06:36.902285 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:06:36.902266    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b\": container with ID starting with 9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b not found: ID does not exist" containerID="9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b"
Sep 09 07:06:36.902376 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.902292    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b"} err="failed to get container status \"9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b\": rpc error: code = NotFound desc = could not find container \"9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b\": container with ID starting with 9141da10ec33f3e54f5ae575226d3b3e4e752d21f2186ee3f5f892217ce5137b not found: ID does not exist"
Sep 09 07:06:36.916686 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.916661    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"]
Sep 09 07:06:36.919256 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:36.919237    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-40c98b/autores-model-40c98b-kserve-55997f5fc9-z6xcd"]
Sep 09 07:06:37.230156 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.230122    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cebb5aa-7615-428c-b34b-00cfcb2b27b0" path="/var/lib/kubelet/pods/6cebb5aa-7615-428c-b34b-00cfcb2b27b0/volumes"
Sep 09 07:06:37.633667 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.633574    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52" podUID="9ddac6af-6e86-44d8-be60-507586dd9291" containerName="main" containerID="cri-o://cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0" gracePeriod=45
Sep 09 07:06:37.886057 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.886000    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:06:37.899161 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.899135    2562 generic.go:356] "Generic (PLEG): container finished" podID="9ddac6af-6e86-44d8-be60-507586dd9291" containerID="cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0" exitCode=0
Sep 09 07:06:37.899475 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.899175    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"
Sep 09 07:06:37.899475 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.899220    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52" event={"ID":"9ddac6af-6e86-44d8-be60-507586dd9291","Type":"ContainerDied","Data":"cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0"}
Sep 09 07:06:37.899475 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.899257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52" event={"ID":"9ddac6af-6e86-44d8-be60-507586dd9291","Type":"ContainerDied","Data":"941056fdae5cfc34bc74e12ab76fbe470f79497448e529dc77ea142417ccb28a"}
Sep 09 07:06:37.899475 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.899281    2562 scope.go:117] "RemoveContainer" containerID="cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0"
Sep 09 07:06:37.909800 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.909776    2562 scope.go:117] "RemoveContainer" containerID="cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0"
Sep 09 07:06:37.910094 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:06:37.910076    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0\": container with ID starting with cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0 not found: ID does not exist" containerID="cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0"
Sep 09 07:06:37.910182 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.910119    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0"} err="failed to get container status \"cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0\": rpc error: code = NotFound desc = could not find container \"cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0\": container with ID starting with cd8f4a44b1c292a0af5506b2e214ebca49957fffaab50975037a5f5cfa188de0 not found: ID does not exist"
Sep 09 07:06:37.946607 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.946586    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-model-cache\") pod \"9ddac6af-6e86-44d8-be60-507586dd9291\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") "
Sep 09 07:06:37.946725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.946615    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ddac6af-6e86-44d8-be60-507586dd9291-tls-certs\") pod \"9ddac6af-6e86-44d8-be60-507586dd9291\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") "
Sep 09 07:06:37.946725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.946678    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-tmp-dir\") pod \"9ddac6af-6e86-44d8-be60-507586dd9291\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") "
Sep 09 07:06:37.946725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.946714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-dshm\") pod \"9ddac6af-6e86-44d8-be60-507586dd9291\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") "
Sep 09 07:06:37.946841 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.946763    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-home\") pod \"9ddac6af-6e86-44d8-be60-507586dd9291\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") "
Sep 09 07:06:37.946841 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.946809    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2g2b\" (UniqueName: \"kubernetes.io/projected/9ddac6af-6e86-44d8-be60-507586dd9291-kube-api-access-b2g2b\") pod \"9ddac6af-6e86-44d8-be60-507586dd9291\" (UID: \"9ddac6af-6e86-44d8-be60-507586dd9291\") "
Sep 09 07:06:37.946919 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.946861    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ddac6af-6e86-44d8-be60-507586dd9291" (UID: "9ddac6af-6e86-44d8-be60-507586dd9291"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:06:37.947015 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.946963    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ddac6af-6e86-44d8-be60-507586dd9291" (UID: "9ddac6af-6e86-44d8-be60-507586dd9291"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:06:37.947058 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.947033    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-home" (OuterVolumeSpecName: "home") pod "9ddac6af-6e86-44d8-be60-507586dd9291" (UID: "9ddac6af-6e86-44d8-be60-507586dd9291"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:06:37.947168 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.947150    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:37.947229 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.947168    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:37.947229 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.947177    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:37.948857 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.948833    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ddac6af-6e86-44d8-be60-507586dd9291-kube-api-access-b2g2b" (OuterVolumeSpecName: "kube-api-access-b2g2b") pod "9ddac6af-6e86-44d8-be60-507586dd9291" (UID: "9ddac6af-6e86-44d8-be60-507586dd9291"). InnerVolumeSpecName "kube-api-access-b2g2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:06:37.948926 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.948857    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-dshm" (OuterVolumeSpecName: "dshm") pod "9ddac6af-6e86-44d8-be60-507586dd9291" (UID: "9ddac6af-6e86-44d8-be60-507586dd9291"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:06:37.948926 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:37.948862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ddac6af-6e86-44d8-be60-507586dd9291-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ddac6af-6e86-44d8-be60-507586dd9291" (UID: "9ddac6af-6e86-44d8-be60-507586dd9291"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:06:38.048523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:38.048486    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ddac6af-6e86-44d8-be60-507586dd9291-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:38.048523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:38.048517    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2g2b\" (UniqueName: \"kubernetes.io/projected/9ddac6af-6e86-44d8-be60-507586dd9291-kube-api-access-b2g2b\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:38.048523 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:38.048528    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ddac6af-6e86-44d8-be60-507586dd9291-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:38.223192 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:38.223160    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"]
Sep 09 07:06:38.225397 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:38.225376    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-8a9acb1e-kserve-54bd8495f9-84v52"]
Sep 09 07:06:39.229461 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:39.229427    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ddac6af-6e86-44d8-be60-507586dd9291" path="/var/lib/kubelet/pods/9ddac6af-6e86-44d8-be60-507586dd9291/volumes"
Sep 09 07:06:41.877519 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:41.877486    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:06:50.328149 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328098    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"]
Sep 09 07:06:50.328621 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328601    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddac6af-6e86-44d8-be60-507586dd9291" containerName="main"
Sep 09 07:06:50.328697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328625    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddac6af-6e86-44d8-be60-507586dd9291" containerName="main"
Sep 09 07:06:50.328697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328649    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cebb5aa-7615-428c-b34b-00cfcb2b27b0" containerName="main"
Sep 09 07:06:50.328697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328658    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cebb5aa-7615-428c-b34b-00cfcb2b27b0" containerName="main"
Sep 09 07:06:50.328697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328672    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76c03cf-6a4b-4995-8063-60f1706d37bb" containerName="revoke-keys"
Sep 09 07:06:50.328697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328680    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76c03cf-6a4b-4995-8063-60f1706d37bb" containerName="revoke-keys"
Sep 09 07:06:50.328940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328776    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cebb5aa-7615-428c-b34b-00cfcb2b27b0" containerName="main"
Sep 09 07:06:50.328940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328791    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ddac6af-6e86-44d8-be60-507586dd9291" containerName="main"
Sep 09 07:06:50.328940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.328802    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76c03cf-6a4b-4995-8063-60f1706d37bb" containerName="revoke-keys"
Sep 09 07:06:50.331890 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.331870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.335820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.335777    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fc9900\"/\"openshift-service-ca.crt\""
Sep 09 07:06:50.335980 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.335830    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fc9900\"/\"default-dockercfg-6pvfj\""
Sep 09 07:06:50.335980 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.335829    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fc9900\"/\"kube-root-ca.crt\""
Sep 09 07:06:50.335980 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.335783    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fc9900\"/\"test-model-fc9900-kserve-self-signed-certs\""
Sep 09 07:06:50.342790 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.342767    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"]
Sep 09 07:06:50.468913 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.468877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2zq\" (UniqueName: \"kubernetes.io/projected/39bbceab-4d11-4afa-9979-12874dfd0c74-kube-api-access-5q2zq\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.469118 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.468921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-home\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.469118 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.468946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bbceab-4d11-4afa-9979-12874dfd0c74-tls-certs\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.469118 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.468970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-tmp-dir\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.469118 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.468987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-model-cache\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.469118 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.469011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-dshm\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.570618 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.570584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-dshm\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.570779 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.570666    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2zq\" (UniqueName: \"kubernetes.io/projected/39bbceab-4d11-4afa-9979-12874dfd0c74-kube-api-access-5q2zq\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.570779 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.570695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-home\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.570779 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.570717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bbceab-4d11-4afa-9979-12874dfd0c74-tls-certs\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.570779 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.570738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-tmp-dir\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.570779 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.570755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-model-cache\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.571134 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.571097    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-model-cache\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.571219 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.571143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-tmp-dir\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.571219 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.571183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-home\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.572846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.572826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-dshm\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.573087 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.573070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bbceab-4d11-4afa-9979-12874dfd0c74-tls-certs\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.578881 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.578813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2zq\" (UniqueName: \"kubernetes.io/projected/39bbceab-4d11-4afa-9979-12874dfd0c74-kube-api-access-5q2zq\") pod \"test-model-fc9900-kserve-59f6f886c5-cwcqg\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") " pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.641113 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.641078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:06:50.917284 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.917251    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"]
Sep 09 07:06:50.917920 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:06:50.917891    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39bbceab_4d11_4afa_9979_12874dfd0c74.slice/crio-196e0ea4f9a8ec51c9aaff7925b44aef5f4482c2d9c3ac8276f8666af5bb6046 WatchSource:0}: Error finding container 196e0ea4f9a8ec51c9aaff7925b44aef5f4482c2d9c3ac8276f8666af5bb6046: Status 404 returned error can't find the container with id 196e0ea4f9a8ec51c9aaff7925b44aef5f4482c2d9c3ac8276f8666af5bb6046
Sep 09 07:06:50.977251 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.977222    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg" event={"ID":"39bbceab-4d11-4afa-9979-12874dfd0c74","Type":"ContainerStarted","Data":"dd34bf311684824b7316b4b1abdb6a40ad0a0c4aa6a5acf21da9e7fa13453598"}
Sep 09 07:06:50.977368 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.977263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg" event={"ID":"39bbceab-4d11-4afa-9979-12874dfd0c74","Type":"ContainerStarted","Data":"196e0ea4f9a8ec51c9aaff7925b44aef5f4482c2d9c3ac8276f8666af5bb6046"}
Sep 09 07:06:50.995253 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:50.995199    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg" podStartSLOduration=0.995178532 podStartE2EDuration="995.178532ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:06:50.99503024 +0000 UTC m=+934.215271364" watchObservedRunningTime="2026-09-09 07:06:50.995178532 +0000 UTC m=+934.215419646"
Sep 09 07:06:54.909066 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:54.909032    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"]
Sep 09 07:06:54.912700 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:54.912677    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"
Sep 09 07:06:54.915486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:54.915460    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:06:54.915603 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:54.915504    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:06:54.921664 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:54.921639    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"]
Sep 09 07:06:55.013692 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:55.013666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073bbc7c-aea7-489e-ae2d-6f10b74b3a9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-29013a28-lp6bk\" (UID: \"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"
Sep 09 07:06:55.114284 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:55.114206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073bbc7c-aea7-489e-ae2d-6f10b74b3a9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-29013a28-lp6bk\" (UID: \"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"
Sep 09 07:06:55.114812 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:55.114794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073bbc7c-aea7-489e-ae2d-6f10b74b3a9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-29013a28-lp6bk\" (UID: \"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"
Sep 09 07:06:55.226044 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:55.226008    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"
Sep 09 07:06:55.496952 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:55.496931    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"]
Sep 09 07:06:55.498458 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:06:55.498434    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073bbc7c_aea7_489e_ae2d_6f10b74b3a9e.slice/crio-582866b89caeda787af37de5c5065aacd31c2775a86d7537ca7485e4f6de1bdf WatchSource:0}: Error finding container 582866b89caeda787af37de5c5065aacd31c2775a86d7537ca7485e4f6de1bdf: Status 404 returned error can't find the container with id 582866b89caeda787af37de5c5065aacd31c2775a86d7537ca7485e4f6de1bdf
Sep 09 07:06:55.777885 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:55.777800    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:06:56.006125 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:56.006072    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk" event={"ID":"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e","Type":"ContainerStarted","Data":"287ddec0f8703da6d43c7f4578fbf71accc65926de9c3e65d881ba5e6382dfc6"}
Sep 09 07:06:56.006508 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:56.006144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk" event={"ID":"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e","Type":"ContainerStarted","Data":"582866b89caeda787af37de5c5065aacd31c2775a86d7537ca7485e4f6de1bdf"}
Sep 09 07:06:56.029460 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:56.029357    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk" podStartSLOduration=2.029342282 podStartE2EDuration="2.029342282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:06:56.027410362 +0000 UTC m=+939.247651474" watchObservedRunningTime="2026-09-09 07:06:56.029342282 +0000 UTC m=+939.249583409"
Sep 09 07:06:57.015274 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:57.015238    2562 generic.go:356] "Generic (PLEG): container finished" podID="073bbc7c-aea7-489e-ae2d-6f10b74b3a9e" containerID="287ddec0f8703da6d43c7f4578fbf71accc65926de9c3e65d881ba5e6382dfc6" exitCode=0
Sep 09 07:06:57.015784 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:57.015286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk" event={"ID":"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e","Type":"ContainerDied","Data":"287ddec0f8703da6d43c7f4578fbf71accc65926de9c3e65d881ba5e6382dfc6"}
Sep 09 07:06:58.154050 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:58.154021    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"
Sep 09 07:06:58.345756 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:58.345658    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073bbc7c-aea7-489e-ae2d-6f10b74b3a9e-maas-api-service-ca\") pod \"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e\" (UID: \"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e\") "
Sep 09 07:06:58.346092 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:58.346061    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073bbc7c-aea7-489e-ae2d-6f10b74b3a9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "073bbc7c-aea7-489e-ae2d-6f10b74b3a9e" (UID: "073bbc7c-aea7-489e-ae2d-6f10b74b3a9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:06:58.446874 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:58.446839    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/073bbc7c-aea7-489e-ae2d-6f10b74b3a9e-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:06:59.028936 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:59.028904    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"
Sep 09 07:06:59.028936 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:59.028910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk" event={"ID":"073bbc7c-aea7-489e-ae2d-6f10b74b3a9e","Type":"ContainerDied","Data":"582866b89caeda787af37de5c5065aacd31c2775a86d7537ca7485e4f6de1bdf"}
Sep 09 07:06:59.029225 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:59.028957    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="582866b89caeda787af37de5c5065aacd31c2775a86d7537ca7485e4f6de1bdf"
Sep 09 07:06:59.883169 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:59.883135    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"]
Sep 09 07:06:59.883560 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:06:59.883392    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy" containerID="cri-o://9f3c809fac9fe93f49c384c1ea7cbdaddf814cb3e13147a24d7832b6814f7365" gracePeriod=30
Sep 09 07:07:00.642093 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:00.642060    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:07:00.642297 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:00.642152    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:07:00.655652 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:00.655628    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:07:01.051115 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:01.051084    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:07:02.453610 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:02.453575    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"]
Sep 09 07:07:02.459749 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:02.459723    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-29013a28-lp6bk"]
Sep 09 07:07:03.230182 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:03.230145    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="073bbc7c-aea7-489e-ae2d-6f10b74b3a9e" path="/var/lib/kubelet/pods/073bbc7c-aea7-489e-ae2d-6f10b74b3a9e/volumes"
Sep 09 07:07:05.071434 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.071359    2562 generic.go:356] "Generic (PLEG): container finished" podID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerID="9f3c809fac9fe93f49c384c1ea7cbdaddf814cb3e13147a24d7832b6814f7365" exitCode=0
Sep 09 07:07:05.071894 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.071554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" event={"ID":"c6c0aefc-35de-4f18-a689-09cd57f6a91e","Type":"ContainerDied","Data":"9f3c809fac9fe93f49c384c1ea7cbdaddf814cb3e13147a24d7832b6814f7365"}
Sep 09 07:07:05.071894 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.071626    2562 scope.go:117] "RemoveContainer" containerID="ea1252d13363a1a9b0fc1511507e654d45ea9053616925c4be8175aa88aac5e4"
Sep 09 07:07:05.126231 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.126208    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:07:05.212030 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.211993    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-socket\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212251 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212036    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-podinfo\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212251 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-envoy\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212251 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212146    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-token\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212251 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212170    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-credential-socket\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212251 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212208    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-data\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212251 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-certs\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212562 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppsdm\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-kube-api-access-ppsdm\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212562 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212334    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:07:05.212562 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istiod-ca-cert\") pod \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\" (UID: \"c6c0aefc-35de-4f18-a689-09cd57f6a91e\") "
Sep 09 07:07:05.212715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212573    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:07:05.212715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212586    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:07:05.212715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212600    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:07:05.212875 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212841    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:05.212875 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212862    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:05.213057 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212876    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:05.213057 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212899    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:05.213057 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.212905    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:07:05.214438 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.214413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:07:05.214570 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.214550    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-kube-api-access-ppsdm" (OuterVolumeSpecName: "kube-api-access-ppsdm") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "kube-api-access-ppsdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:07:05.214833 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.214805    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:07:05.214833 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.214818    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6c0aefc-35de-4f18-a689-09cd57f6a91e" (UID: "c6c0aefc-35de-4f18-a689-09cd57f6a91e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:07:05.313613 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.313581    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:05.313613 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.313608    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppsdm\" (UniqueName: \"kubernetes.io/projected/c6c0aefc-35de-4f18-a689-09cd57f6a91e-kube-api-access-ppsdm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:05.313613 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.313617    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:05.313825 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.313627    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:05.313825 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:05.313636    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c0aefc-35de-4f18-a689-09cd57f6a91e-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:06.078373 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:06.078346    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"
Sep 09 07:07:06.078845 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:06.078378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c" event={"ID":"c6c0aefc-35de-4f18-a689-09cd57f6a91e","Type":"ContainerDied","Data":"6c33b3d72d4530f01d8caa6a35a6d67446e5ba75c7356733feeea7b6c413572c"}
Sep 09 07:07:06.078845 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:06.078437    2562 scope.go:117] "RemoveContainer" containerID="9f3c809fac9fe93f49c384c1ea7cbdaddf814cb3e13147a24d7832b6814f7365"
Sep 09 07:07:06.097677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:06.097652    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"]
Sep 09 07:07:06.101983 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:06.101958    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-40c98b-openshift-default-748fc9745d-nkw7c"]
Sep 09 07:07:07.229900 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:07.229866    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" path="/var/lib/kubelet/pods/c6c0aefc-35de-4f18-a689-09cd57f6a91e/volumes"
Sep 09 07:07:13.377430 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:13.377394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:07:27.777339 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:27.777303    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:07:41.770579 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:41.770499    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:07:48.834503 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.834470    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"]
Sep 09 07:07:48.834878 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.834796    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073bbc7c-aea7-489e-ae2d-6f10b74b3a9e" containerName="revoke-keys"
Sep 09 07:07:48.834878 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.834807    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="073bbc7c-aea7-489e-ae2d-6f10b74b3a9e" containerName="revoke-keys"
Sep 09 07:07:48.834878 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.834819    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy"
Sep 09 07:07:48.834878 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.834825    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy"
Sep 09 07:07:48.835034 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.834887    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="073bbc7c-aea7-489e-ae2d-6f10b74b3a9e" containerName="revoke-keys"
Sep 09 07:07:48.835034 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.834895    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy"
Sep 09 07:07:48.835034 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.834900    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy"
Sep 09 07:07:48.837748 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.837731    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.840528 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.840504    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a1a33e\"/\"kube-root-ca.crt\""
Sep 09 07:07:48.840651 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.840574    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a1a33e\"/\"default-dockercfg-bpzc2\""
Sep 09 07:07:48.841345 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.841326    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a1a33e\"/\"openshift-service-ca.crt\""
Sep 09 07:07:48.841433 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.841369    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a1a33e\"/\"explicit-model-a1a33e-kserve-self-signed-certs\""
Sep 09 07:07:48.849714 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.849693    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"]
Sep 09 07:07:48.893169 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.893138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wdm\" (UniqueName: \"kubernetes.io/projected/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-kube-api-access-j7wdm\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.893320 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.893177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tls-certs\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.893320 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.893197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-dshm\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.893320 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.893277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-model-cache\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.893435 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.893355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-home\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.893435 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.893379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tmp-dir\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994176 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tmp-dir\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994323 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wdm\" (UniqueName: \"kubernetes.io/projected/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-kube-api-access-j7wdm\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994323 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tls-certs\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994407 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-dshm\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994407 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-model-cache\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994524 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-home\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994581 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994536    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tmp-dir\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994757 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-model-cache\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.994829 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.994790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-home\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.996561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.996539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-dshm\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:48.996674 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:48.996657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tls-certs\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:49.003209 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:49.003175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wdm\" (UniqueName: \"kubernetes.io/projected/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-kube-api-access-j7wdm\") pod \"explicit-model-a1a33e-kserve-6b9d888b69-2cdtp\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") " pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:49.147397 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:49.147318    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:49.426291 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:49.426263    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"]
Sep 09 07:07:49.427883 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:07:49.427859    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29923c3d_12cd_4f5a_b815_3e5bd0dd34e6.slice/crio-8316af43a8ffcd4ca73d0a9fb4b4c141d8ac8b8581992e40e6de497d9b73eabd WatchSource:0}: Error finding container 8316af43a8ffcd4ca73d0a9fb4b4c141d8ac8b8581992e40e6de497d9b73eabd: Status 404 returned error can't find the container with id 8316af43a8ffcd4ca73d0a9fb4b4c141d8ac8b8581992e40e6de497d9b73eabd
Sep 09 07:07:50.317652 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:50.317613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp" event={"ID":"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6","Type":"ContainerStarted","Data":"9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c"}
Sep 09 07:07:50.317652 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:50.317652    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp" event={"ID":"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6","Type":"ContainerStarted","Data":"8316af43a8ffcd4ca73d0a9fb4b4c141d8ac8b8581992e40e6de497d9b73eabd"}
Sep 09 07:07:50.341782 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:50.341730    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp" podStartSLOduration=2.341714468 podStartE2EDuration="2.341714468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:07:50.339411226 +0000 UTC m=+993.559652340" watchObservedRunningTime="2026-09-09 07:07:50.341714468 +0000 UTC m=+993.561955581"
Sep 09 07:07:52.212851 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.212818    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"]
Sep 09 07:07:52.369378 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.369344    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"]
Sep 09 07:07:52.369738 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.369718    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy"
Sep 09 07:07:52.369738 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.369737    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0aefc-35de-4f18-a689-09cd57f6a91e" containerName="istio-proxy"
Sep 09 07:07:52.372881 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.372865    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"
Sep 09 07:07:52.376535 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.376515    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:07:52.376643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.376556    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:07:52.383220 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.383197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"]
Sep 09 07:07:52.425008 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.424980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd34be8-fbfd-465d-96ca-d241274210c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q\" (UID: \"7cd34be8-fbfd-465d-96ca-d241274210c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"
Sep 09 07:07:52.526366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.526296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd34be8-fbfd-465d-96ca-d241274210c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q\" (UID: \"7cd34be8-fbfd-465d-96ca-d241274210c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"
Sep 09 07:07:52.526890 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.526871    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd34be8-fbfd-465d-96ca-d241274210c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q\" (UID: \"7cd34be8-fbfd-465d-96ca-d241274210c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"
Sep 09 07:07:52.684793 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.684747    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"
Sep 09 07:07:52.956789 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:52.956756    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"]
Sep 09 07:07:52.957402 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:07:52.957376    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd34be8_fbfd_465d_96ca_d241274210c6.slice/crio-ef41ba5c6dc7cd57e0017ab148f11f78c82b28437b2ac30b6b6c84ae653a823b WatchSource:0}: Error finding container ef41ba5c6dc7cd57e0017ab148f11f78c82b28437b2ac30b6b6c84ae653a823b: Status 404 returned error can't find the container with id ef41ba5c6dc7cd57e0017ab148f11f78c82b28437b2ac30b6b6c84ae653a823b
Sep 09 07:07:53.335097 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:53.335063    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q" event={"ID":"7cd34be8-fbfd-465d-96ca-d241274210c6","Type":"ContainerStarted","Data":"00c13d45cf35bc3c2473923e671459c570a5038fcfe7f4baac432dd4f430c2b4"}
Sep 09 07:07:53.335450 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:53.335129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q" event={"ID":"7cd34be8-fbfd-465d-96ca-d241274210c6","Type":"ContainerStarted","Data":"ef41ba5c6dc7cd57e0017ab148f11f78c82b28437b2ac30b6b6c84ae653a823b"}
Sep 09 07:07:53.352765 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:53.352723    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q" podStartSLOduration=1.352707885 podStartE2EDuration="1.352707885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:07:53.350710418 +0000 UTC m=+996.570951531" watchObservedRunningTime="2026-09-09 07:07:53.352707885 +0000 UTC m=+996.572948999"
Sep 09 07:07:54.342173 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:54.342139    2562 generic.go:356] "Generic (PLEG): container finished" podID="7cd34be8-fbfd-465d-96ca-d241274210c6" containerID="00c13d45cf35bc3c2473923e671459c570a5038fcfe7f4baac432dd4f430c2b4" exitCode=0
Sep 09 07:07:54.342573 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:54.342224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q" event={"ID":"7cd34be8-fbfd-465d-96ca-d241274210c6","Type":"ContainerDied","Data":"00c13d45cf35bc3c2473923e671459c570a5038fcfe7f4baac432dd4f430c2b4"}
Sep 09 07:07:55.475771 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:55.475748    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"
Sep 09 07:07:55.552491 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:55.552461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd34be8-fbfd-465d-96ca-d241274210c6-maas-api-service-ca\") pod \"7cd34be8-fbfd-465d-96ca-d241274210c6\" (UID: \"7cd34be8-fbfd-465d-96ca-d241274210c6\") "
Sep 09 07:07:55.552840 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:55.552819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd34be8-fbfd-465d-96ca-d241274210c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7cd34be8-fbfd-465d-96ca-d241274210c6" (UID: "7cd34be8-fbfd-465d-96ca-d241274210c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:07:55.654050 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:55.653956    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd34be8-fbfd-465d-96ca-d241274210c6-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:07:55.961887 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:55.961845    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:07:56.354506 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:56.354419    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"
Sep 09 07:07:56.354506 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:56.354445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q" event={"ID":"7cd34be8-fbfd-465d-96ca-d241274210c6","Type":"ContainerDied","Data":"ef41ba5c6dc7cd57e0017ab148f11f78c82b28437b2ac30b6b6c84ae653a823b"}
Sep 09 07:07:56.354506 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:56.354496    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef41ba5c6dc7cd57e0017ab148f11f78c82b28437b2ac30b6b6c84ae653a823b"
Sep 09 07:07:57.317340 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.317301    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"]
Sep 09 07:07:57.471762 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.471726    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"]
Sep 09 07:07:57.472192 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.472173    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd34be8-fbfd-465d-96ca-d241274210c6" containerName="revoke-keys"
Sep 09 07:07:57.472192 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.472191    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd34be8-fbfd-465d-96ca-d241274210c6" containerName="revoke-keys"
Sep 09 07:07:57.472347 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.472282    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd34be8-fbfd-465d-96ca-d241274210c6" containerName="revoke-keys"
Sep 09 07:07:57.475561 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.475536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"
Sep 09 07:07:57.478750 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.478722    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:07:57.478869 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.478846    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:07:57.490267 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.490240    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"]
Sep 09 07:07:57.572481 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.572398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244648cf-07b1-4f8a-ba2a-07cdd9f57031-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b212cb-jc447\" (UID: \"244648cf-07b1-4f8a-ba2a-07cdd9f57031\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"
Sep 09 07:07:57.673893 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.673860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244648cf-07b1-4f8a-ba2a-07cdd9f57031-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b212cb-jc447\" (UID: \"244648cf-07b1-4f8a-ba2a-07cdd9f57031\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"
Sep 09 07:07:57.674528 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.674505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244648cf-07b1-4f8a-ba2a-07cdd9f57031-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b212cb-jc447\" (UID: \"244648cf-07b1-4f8a-ba2a-07cdd9f57031\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"
Sep 09 07:07:57.790860 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:57.790827    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"
Sep 09 07:07:58.094131 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:58.093081    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"]
Sep 09 07:07:58.098898 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:07:58.098866    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244648cf_07b1_4f8a_ba2a_07cdd9f57031.slice/crio-cdfe45fa2a21fd568ec807ff0cdbd22c69d839ac5be8ac5b730ec43e22e50285 WatchSource:0}: Error finding container cdfe45fa2a21fd568ec807ff0cdbd22c69d839ac5be8ac5b730ec43e22e50285: Status 404 returned error can't find the container with id cdfe45fa2a21fd568ec807ff0cdbd22c69d839ac5be8ac5b730ec43e22e50285
Sep 09 07:07:58.368863 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:58.368775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447" event={"ID":"244648cf-07b1-4f8a-ba2a-07cdd9f57031","Type":"ContainerStarted","Data":"cdfe45fa2a21fd568ec807ff0cdbd22c69d839ac5be8ac5b730ec43e22e50285"}
Sep 09 07:07:59.147666 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:59.147588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:07:59.376188 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:59.376147    2562 generic.go:356] "Generic (PLEG): container finished" podID="244648cf-07b1-4f8a-ba2a-07cdd9f57031" containerID="3c7fe9d288f6f7b2124d03bac3fabcf2d8f4af898d923d7a0ae7b3c40faf928d" exitCode=0
Sep 09 07:07:59.376571 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:07:59.376233    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447" event={"ID":"244648cf-07b1-4f8a-ba2a-07cdd9f57031","Type":"ContainerDied","Data":"3c7fe9d288f6f7b2124d03bac3fabcf2d8f4af898d923d7a0ae7b3c40faf928d"}
Sep 09 07:08:00.512180 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:00.512155    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"
Sep 09 07:08:00.600470 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:00.600427    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244648cf-07b1-4f8a-ba2a-07cdd9f57031-maas-api-service-ca\") pod \"244648cf-07b1-4f8a-ba2a-07cdd9f57031\" (UID: \"244648cf-07b1-4f8a-ba2a-07cdd9f57031\") "
Sep 09 07:08:00.600838 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:00.600816    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244648cf-07b1-4f8a-ba2a-07cdd9f57031-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "244648cf-07b1-4f8a-ba2a-07cdd9f57031" (UID: "244648cf-07b1-4f8a-ba2a-07cdd9f57031"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:08:00.701182 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:00.701158    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244648cf-07b1-4f8a-ba2a-07cdd9f57031-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:01.390236 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:01.390203    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"
Sep 09 07:08:01.390396 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:01.390244    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447" event={"ID":"244648cf-07b1-4f8a-ba2a-07cdd9f57031","Type":"ContainerDied","Data":"cdfe45fa2a21fd568ec807ff0cdbd22c69d839ac5be8ac5b730ec43e22e50285"}
Sep 09 07:08:01.390396 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:01.390299    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdfe45fa2a21fd568ec807ff0cdbd22c69d839ac5be8ac5b730ec43e22e50285"
Sep 09 07:08:03.407808 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:03.407766    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"]
Sep 09 07:08:03.412686 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:03.412655    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a33e-l4f6q"]
Sep 09 07:08:05.229988 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:05.229955    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd34be8-fbfd-465d-96ca-d241274210c6" path="/var/lib/kubelet/pods/7cd34be8-fbfd-465d-96ca-d241274210c6/volumes"
Sep 09 07:08:07.347818 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.347748    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp" podUID="29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" containerName="main" containerID="cri-o://9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c" gracePeriod=45
Sep 09 07:08:07.600570 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.600517    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:08:07.667915 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.667887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-model-cache\") pod \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") "
Sep 09 07:08:07.668113 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.667979    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7wdm\" (UniqueName: \"kubernetes.io/projected/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-kube-api-access-j7wdm\") pod \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") "
Sep 09 07:08:07.668113 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668041    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-dshm\") pod \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") "
Sep 09 07:08:07.668113 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668066    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-home\") pod \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") "
Sep 09 07:08:07.668303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668174    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" (UID: "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:07.668303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668180    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tmp-dir\") pod \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") "
Sep 09 07:08:07.668303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668271    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tls-certs\") pod \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\" (UID: \"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6\") "
Sep 09 07:08:07.668471 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668391    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" (UID: "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:07.668471 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-home" (OuterVolumeSpecName: "home") pod "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" (UID: "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:07.668663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668644    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:07.668720 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668671    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:07.668720 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.668687    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:07.670381 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.670359    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-dshm" (OuterVolumeSpecName: "dshm") pod "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" (UID: "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:07.670639 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.670615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" (UID: "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:08:07.670702 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.670636    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-kube-api-access-j7wdm" (OuterVolumeSpecName: "kube-api-access-j7wdm") pod "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" (UID: "29923c3d-12cd-4f5a-b815-3e5bd0dd34e6"). InnerVolumeSpecName "kube-api-access-j7wdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:08:07.675713 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.675691    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:08:07.770066 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.770030    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7wdm\" (UniqueName: \"kubernetes.io/projected/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-kube-api-access-j7wdm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:07.770066 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.770064    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:07.770299 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:07.770080    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:08.431868 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.431837    2562 generic.go:356] "Generic (PLEG): container finished" podID="29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" containerID="9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c" exitCode=0
Sep 09 07:08:08.432381 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.431883    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"
Sep 09 07:08:08.432381 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.431903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp" event={"ID":"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6","Type":"ContainerDied","Data":"9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c"}
Sep 09 07:08:08.432381 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.431934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp" event={"ID":"29923c3d-12cd-4f5a-b815-3e5bd0dd34e6","Type":"ContainerDied","Data":"8316af43a8ffcd4ca73d0a9fb4b4c141d8ac8b8581992e40e6de497d9b73eabd"}
Sep 09 07:08:08.432381 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.431951    2562 scope.go:117] "RemoveContainer" containerID="9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c"
Sep 09 07:08:08.444714 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.444678    2562 scope.go:117] "RemoveContainer" containerID="9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c"
Sep 09 07:08:08.445435 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:08:08.445397    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c\": container with ID starting with 9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c not found: ID does not exist" containerID="9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c"
Sep 09 07:08:08.445557 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.445446    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c"} err="failed to get container status \"9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c\": rpc error: code = NotFound desc = could not find container \"9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c\": container with ID starting with 9011b07f514cfda8fc266457c2babe195985b93668af61e132621949f278aa8c not found: ID does not exist"
Sep 09 07:08:08.458629 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.458604    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"]
Sep 09 07:08:08.464538 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.464513    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-a1a33e/explicit-model-a1a33e-kserve-6b9d888b69-2cdtp"]
Sep 09 07:08:08.525465 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.525441    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"]
Sep 09 07:08:08.530008 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:08.529988    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b212cb-jc447"]
Sep 09 07:08:09.229905 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:09.229871    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="244648cf-07b1-4f8a-ba2a-07cdd9f57031" path="/var/lib/kubelet/pods/244648cf-07b1-4f8a-ba2a-07cdd9f57031/volumes"
Sep 09 07:08:09.230299 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:09.230285    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" path="/var/lib/kubelet/pods/29923c3d-12cd-4f5a-b815-3e5bd0dd34e6/volumes"
Sep 09 07:08:12.337677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.337610    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg" podUID="39bbceab-4d11-4afa-9979-12874dfd0c74" containerName="main" containerID="cri-o://dd34bf311684824b7316b4b1abdb6a40ad0a0c4aa6a5acf21da9e7fa13453598" gracePeriod=45
Sep 09 07:08:12.458030 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.458003    2562 generic.go:356] "Generic (PLEG): container finished" podID="39bbceab-4d11-4afa-9979-12874dfd0c74" containerID="dd34bf311684824b7316b4b1abdb6a40ad0a0c4aa6a5acf21da9e7fa13453598" exitCode=0
Sep 09 07:08:12.458181 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.458080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg" event={"ID":"39bbceab-4d11-4afa-9979-12874dfd0c74","Type":"ContainerDied","Data":"dd34bf311684824b7316b4b1abdb6a40ad0a0c4aa6a5acf21da9e7fa13453598"}
Sep 09 07:08:12.588871 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.588814    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:08:12.716850 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.716820    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q2zq\" (UniqueName: \"kubernetes.io/projected/39bbceab-4d11-4afa-9979-12874dfd0c74-kube-api-access-5q2zq\") pod \"39bbceab-4d11-4afa-9979-12874dfd0c74\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") "
Sep 09 07:08:12.717043 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.716863    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bbceab-4d11-4afa-9979-12874dfd0c74-tls-certs\") pod \"39bbceab-4d11-4afa-9979-12874dfd0c74\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") "
Sep 09 07:08:12.717043 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.716905    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-tmp-dir\") pod \"39bbceab-4d11-4afa-9979-12874dfd0c74\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") "
Sep 09 07:08:12.717043 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.716920    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-dshm\") pod \"39bbceab-4d11-4afa-9979-12874dfd0c74\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") "
Sep 09 07:08:12.717043 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.716958    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-home\") pod \"39bbceab-4d11-4afa-9979-12874dfd0c74\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") "
Sep 09 07:08:12.717043 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.716986    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-model-cache\") pod \"39bbceab-4d11-4afa-9979-12874dfd0c74\" (UID: \"39bbceab-4d11-4afa-9979-12874dfd0c74\") "
Sep 09 07:08:12.717345 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.717206    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39bbceab-4d11-4afa-9979-12874dfd0c74" (UID: "39bbceab-4d11-4afa-9979-12874dfd0c74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:12.717345 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.717228    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-home" (OuterVolumeSpecName: "home") pod "39bbceab-4d11-4afa-9979-12874dfd0c74" (UID: "39bbceab-4d11-4afa-9979-12874dfd0c74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:12.717345 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.717315    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-model-cache" (OuterVolumeSpecName: "model-cache") pod "39bbceab-4d11-4afa-9979-12874dfd0c74" (UID: "39bbceab-4d11-4afa-9979-12874dfd0c74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:12.717506 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.717392    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:12.717506 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.717418    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:12.719231 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.719203    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bbceab-4d11-4afa-9979-12874dfd0c74-kube-api-access-5q2zq" (OuterVolumeSpecName: "kube-api-access-5q2zq") pod "39bbceab-4d11-4afa-9979-12874dfd0c74" (UID: "39bbceab-4d11-4afa-9979-12874dfd0c74"). InnerVolumeSpecName "kube-api-access-5q2zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:08:12.719231 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.719212    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-dshm" (OuterVolumeSpecName: "dshm") pod "39bbceab-4d11-4afa-9979-12874dfd0c74" (UID: "39bbceab-4d11-4afa-9979-12874dfd0c74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:12.719427 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.719236    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bbceab-4d11-4afa-9979-12874dfd0c74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39bbceab-4d11-4afa-9979-12874dfd0c74" (UID: "39bbceab-4d11-4afa-9979-12874dfd0c74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:08:12.818743 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.818702    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:12.818743 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.818744    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bbceab-4d11-4afa-9979-12874dfd0c74-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:12.819011 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.818761    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q2zq\" (UniqueName: \"kubernetes.io/projected/39bbceab-4d11-4afa-9979-12874dfd0c74-kube-api-access-5q2zq\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:12.819011 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:12.818776    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bbceab-4d11-4afa-9979-12874dfd0c74-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:13.464846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:13.464815    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"
Sep 09 07:08:13.464846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:13.464817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg" event={"ID":"39bbceab-4d11-4afa-9979-12874dfd0c74","Type":"ContainerDied","Data":"196e0ea4f9a8ec51c9aaff7925b44aef5f4482c2d9c3ac8276f8666af5bb6046"}
Sep 09 07:08:13.465478 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:13.464870    2562 scope.go:117] "RemoveContainer" containerID="dd34bf311684824b7316b4b1abdb6a40ad0a0c4aa6a5acf21da9e7fa13453598"
Sep 09 07:08:13.482642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:13.482618    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"]
Sep 09 07:08:13.486372 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:13.486347    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fc9900/test-model-fc9900-kserve-59f6f886c5-cwcqg"]
Sep 09 07:08:15.229668 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:15.229634    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39bbceab-4d11-4afa-9979-12874dfd0c74" path="/var/lib/kubelet/pods/39bbceab-4d11-4afa-9979-12874dfd0c74/volumes"
Sep 09 07:08:15.401529 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:15.401491    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"]
Sep 09 07:08:15.401764 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:15.401739    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy" containerID="cri-o://24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a" gracePeriod=30
Sep 09 07:08:17.080250 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:17.080213    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:08:18.965427 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:18.965380    2562 patch_prober.go:28] interesting pod/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 07:08:18.965830 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:18.965447    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 07:08:20.654475 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.654447    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:08:20.785970 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.785933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-envoy\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.785970 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.785973    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-data\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.786206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786003    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004450e9-6a47-4f78-9b14-f355b707fd28-istiod-ca-cert\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.786206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786072    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dptkt\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-kube-api-access-dptkt\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.786206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786096    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-credential-socket\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.786206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786134    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-socket\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.786206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-certs\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.786206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786179    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-istio-token\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.786514 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786252    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004450e9-6a47-4f78-9b14-f355b707fd28-istio-podinfo\") pod \"004450e9-6a47-4f78-9b14-f355b707fd28\" (UID: \"004450e9-6a47-4f78-9b14-f355b707fd28\") "
Sep 09 07:08:20.786514 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786322    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-data" (OuterVolumeSpecName: "istio-data") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:20.786514 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786442    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:20.786666 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786562    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:20.786666 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786618    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:20.786666 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786448    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004450e9-6a47-4f78-9b14-f355b707fd28-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:08:20.786814 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786682    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:20.786814 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.786735    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:20.788346 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.788318    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-istio-token" (OuterVolumeSpecName: "istio-token") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:08:20.788462 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.788419    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/004450e9-6a47-4f78-9b14-f355b707fd28-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:08:20.788604 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.788585    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:20.788682 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.788663    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-kube-api-access-dptkt" (OuterVolumeSpecName: "kube-api-access-dptkt") pod "004450e9-6a47-4f78-9b14-f355b707fd28" (UID: "004450e9-6a47-4f78-9b14-f355b707fd28"). InnerVolumeSpecName "kube-api-access-dptkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:08:20.888011 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.887976    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptkt\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-kube-api-access-dptkt\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:20.888011 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.888005    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:20.888011 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.888014    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:20.888250 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.888023    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004450e9-6a47-4f78-9b14-f355b707fd28-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:20.888250 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.888034    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004450e9-6a47-4f78-9b14-f355b707fd28-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:20.888250 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.888041    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004450e9-6a47-4f78-9b14-f355b707fd28-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:20.888250 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:20.888048    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004450e9-6a47-4f78-9b14-f355b707fd28-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:21.512843 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.512750    2562 generic.go:356] "Generic (PLEG): container finished" podID="004450e9-6a47-4f78-9b14-f355b707fd28" containerID="24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a" exitCode=0
Sep 09 07:08:21.512843 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.512803    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"
Sep 09 07:08:21.512843 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.512830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" event={"ID":"004450e9-6a47-4f78-9b14-f355b707fd28","Type":"ContainerDied","Data":"24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a"}
Sep 09 07:08:21.513093 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.512888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq" event={"ID":"004450e9-6a47-4f78-9b14-f355b707fd28","Type":"ContainerDied","Data":"cce5491f977d8f27fda2ff669be67f3513d80ef5d802c017d9fdd9bf2be43256"}
Sep 09 07:08:21.513093 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.512910    2562 scope.go:117] "RemoveContainer" containerID="24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a"
Sep 09 07:08:21.521893 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.521710    2562 scope.go:117] "RemoveContainer" containerID="d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3"
Sep 09 07:08:21.529562 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.529544    2562 scope.go:117] "RemoveContainer" containerID="24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a"
Sep 09 07:08:21.529816 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:08:21.529800    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a\": container with ID starting with 24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a not found: ID does not exist" containerID="24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a"
Sep 09 07:08:21.529864 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.529823    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a"} err="failed to get container status \"24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a\": rpc error: code = NotFound desc = could not find container \"24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a\": container with ID starting with 24f4bed61008567790cd897f0eeb1a593ee71fcd440381d716b6026dfc13652a not found: ID does not exist"
Sep 09 07:08:21.529864 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.529837    2562 scope.go:117] "RemoveContainer" containerID="d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3"
Sep 09 07:08:21.530051 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:08:21.530034    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3\": container with ID starting with d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3 not found: ID does not exist" containerID="d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3"
Sep 09 07:08:21.530241 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.530056    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3"} err="failed to get container status \"d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3\": rpc error: code = NotFound desc = could not find container \"d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3\": container with ID starting with d9f37d3aaec3d65260e46a45f9336e86f4414c4a4a3621ff3058b706a61828d3 not found: ID does not exist"
Sep 09 07:08:21.532839 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.532819    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"]
Sep 09 07:08:21.537352 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:21.537327    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b212cb-openshift-default-bc4bdbbb6-26sgq"]
Sep 09 07:08:23.230329 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:23.230295    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" path="/var/lib/kubelet/pods/004450e9-6a47-4f78-9b14-f355b707fd28/volumes"
Sep 09 07:08:27.970814 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:27.970778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:08:33.782567 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:33.782531    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"]
Sep 09 07:08:33.783129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:33.782797    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy" containerID="cri-o://3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737" gracePeriod=30
Sep 09 07:08:35.922971 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:35.922929    2562 patch_prober.go:28] interesting pod/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 07:08:35.923378 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:35.923005    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 07:08:36.472705 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:36.472672    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:08:39.025813 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.025786    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:08:39.050961 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.050899    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-credential-socket\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.050961 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.050941    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-socket\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.051162 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051005    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a57d65-72e8-4306-9105-228095c56581-istio-podinfo\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.051162 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051030    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc4xc\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-kube-api-access-tc4xc\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.051162 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051060    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-istio-token\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.051162 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-certs\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.051367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051182    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-envoy\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.051367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051213    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a57d65-72e8-4306-9105-228095c56581-istiod-ca-cert\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.051367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051206    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:39.051367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051317    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-data\") pod \"f4a57d65-72e8-4306-9105-228095c56581\" (UID: \"f4a57d65-72e8-4306-9105-228095c56581\") "
Sep 09 07:08:39.051367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051330    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:39.051634 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051489    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:39.051634 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:39.051744 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051635    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a57d65-72e8-4306-9105-228095c56581-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:08:39.051744 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051697    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a57d65-72e8-4306-9105-228095c56581-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.051744 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051711    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.051744 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051724    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.051744 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051737    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.052001 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.051751    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.053410 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.053381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4a57d65-72e8-4306-9105-228095c56581-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:08:39.053515 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.053377    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-kube-api-access-tc4xc" (OuterVolumeSpecName: "kube-api-access-tc4xc") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "kube-api-access-tc4xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:08:39.053515 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.053458    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:08:39.053642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.053625    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4a57d65-72e8-4306-9105-228095c56581" (UID: "f4a57d65-72e8-4306-9105-228095c56581"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:08:39.152484 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.152450    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a57d65-72e8-4306-9105-228095c56581-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.152484 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.152481    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4xc\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-kube-api-access-tc4xc\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.152484 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.152491    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a57d65-72e8-4306-9105-228095c56581-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.152783 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.152499    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a57d65-72e8-4306-9105-228095c56581-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:08:39.615206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.615170    2562 generic.go:356] "Generic (PLEG): container finished" podID="f4a57d65-72e8-4306-9105-228095c56581" containerID="3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737" exitCode=0
Sep 09 07:08:39.615390 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.615218    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"
Sep 09 07:08:39.615390 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.615303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" event={"ID":"f4a57d65-72e8-4306-9105-228095c56581","Type":"ContainerDied","Data":"3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737"}
Sep 09 07:08:39.615390 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.615351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5" event={"ID":"f4a57d65-72e8-4306-9105-228095c56581","Type":"ContainerDied","Data":"864c708a10eb8805f207e19953b5712a95e34a7b1a52ff58bd2fa82d2ad3f457"}
Sep 09 07:08:39.615390 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.615378    2562 scope.go:117] "RemoveContainer" containerID="3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737"
Sep 09 07:08:39.625366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.625332    2562 scope.go:117] "RemoveContainer" containerID="baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17"
Sep 09 07:08:39.633879 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.633858    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"]
Sep 09 07:08:39.634725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.634701    2562 scope.go:117] "RemoveContainer" containerID="3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737"
Sep 09 07:08:39.635090 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:08:39.635059    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737\": container with ID starting with 3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737 not found: ID does not exist" containerID="3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737"
Sep 09 07:08:39.635090 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.635120    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737"} err="failed to get container status \"3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737\": rpc error: code = NotFound desc = could not find container \"3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737\": container with ID starting with 3da0cf83177c1e466d552b5185446dd1fe4543e387fe9244c9956ac62e83b737 not found: ID does not exist"
Sep 09 07:08:39.635090 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.635147    2562 scope.go:117] "RemoveContainer" containerID="baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17"
Sep 09 07:08:39.635510 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:08:39.635484    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17\": container with ID starting with baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17 not found: ID does not exist" containerID="baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17"
Sep 09 07:08:39.635614 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.635517    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17"} err="failed to get container status \"baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17\": rpc error: code = NotFound desc = could not find container \"baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17\": container with ID starting with baf78523d4263bea43753bb447e80f35b32d169d1dcb2d3009ccec735c8dba17 not found: ID does not exist"
Sep 09 07:08:39.641468 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:39.641445    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fc9900-openshift-default-6cf679bf54-clfx5"]
Sep 09 07:08:40.726730 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.726691    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"]
Sep 09 07:08:40.727123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727029    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy"
Sep 09 07:08:40.727123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727041    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy"
Sep 09 07:08:40.727123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727052    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy"
Sep 09 07:08:40.727123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727061    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy"
Sep 09 07:08:40.727123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727071    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244648cf-07b1-4f8a-ba2a-07cdd9f57031" containerName="revoke-keys"
Sep 09 07:08:40.727123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727078    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="244648cf-07b1-4f8a-ba2a-07cdd9f57031" containerName="revoke-keys"
Sep 09 07:08:40.727123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727083    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy"
Sep 09 07:08:40.727123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727088    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727130    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39bbceab-4d11-4afa-9979-12874dfd0c74" containerName="main"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727139    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39bbceab-4d11-4afa-9979-12874dfd0c74" containerName="main"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727152    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" containerName="main"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727157    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" containerName="main"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727214    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727225    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727232    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a57d65-72e8-4306-9105-228095c56581" containerName="istio-proxy"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727237    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="244648cf-07b1-4f8a-ba2a-07cdd9f57031" containerName="revoke-keys"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727244    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="29923c3d-12cd-4f5a-b815-3e5bd0dd34e6" containerName="main"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727250    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39bbceab-4d11-4afa-9979-12874dfd0c74" containerName="main"
Sep 09 07:08:40.727384 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.727256    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy"
Sep 09 07:08:40.731872 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.731841    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.734806 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.734783    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c20824ba-openshift-default-dockercfg-42cvp\""
Sep 09 07:08:40.744815 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.744790    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"]
Sep 09 07:08:40.766335 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766309    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffcdecb-ae89-45b2-a783-b637189a43ca-istiod-ca-cert\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.766583 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-data\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.766696 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-token\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.766696 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-credential-socket\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.766696 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvj7l\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-kube-api-access-hvj7l\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.766877 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-podinfo\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.766877 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-socket\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.766877 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-envoy\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.766877 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.766853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-certs\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868120 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffcdecb-ae89-45b2-a783-b637189a43ca-istiod-ca-cert\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868329 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-data\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868329 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-token\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868329 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-credential-socket\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868329 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvj7l\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-kube-api-access-hvj7l\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868329 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-podinfo\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868619 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-socket\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868619 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-envoy\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868619 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-certs\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868772 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-credential-socket\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868825 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-socket\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868825 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-certs\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.868942 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.868823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-data\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.869228 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.869208    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffcdecb-ae89-45b2-a783-b637189a43ca-istiod-ca-cert\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.870899 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.870877    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-envoy\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.871026 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.870963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-podinfo\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.878675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.878653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvj7l\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-kube-api-access-hvj7l\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:40.878794 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:40.878775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-token\") pod \"e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") " pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:41.043802 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.043717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:41.229930 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.229900    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a57d65-72e8-4306-9105-228095c56581" path="/var/lib/kubelet/pods/f4a57d65-72e8-4306-9105-228095c56581/volumes"
Sep 09 07:08:41.331393 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.331371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"]
Sep 09 07:08:41.333163 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:08:41.333137    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffcdecb_ae89_45b2_a783_b637189a43ca.slice/crio-63ac3bbe4f43aa7e1d7327287c910398b5d4231c17be1619cbb34a65f86907b2 WatchSource:0}: Error finding container 63ac3bbe4f43aa7e1d7327287c910398b5d4231c17be1619cbb34a65f86907b2: Status 404 returned error can't find the container with id 63ac3bbe4f43aa7e1d7327287c910398b5d4231c17be1619cbb34a65f86907b2
Sep 09 07:08:41.335438 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.335403    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:08:41.335504 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.335474    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:08:41.335543 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.335503    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:08:41.629920 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.629836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" event={"ID":"3ffcdecb-ae89-45b2-a783-b637189a43ca","Type":"ContainerStarted","Data":"c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2"}
Sep 09 07:08:41.629920 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.629894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" event={"ID":"3ffcdecb-ae89-45b2-a783-b637189a43ca","Type":"ContainerStarted","Data":"63ac3bbe4f43aa7e1d7327287c910398b5d4231c17be1619cbb34a65f86907b2"}
Sep 09 07:08:41.651730 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:41.651676    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podStartSLOduration=1.6516590789999999 podStartE2EDuration="1.651659079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:08:41.649836219 +0000 UTC m=+1044.870077334" watchObservedRunningTime="2026-09-09 07:08:41.651659079 +0000 UTC m=+1044.871900193"
Sep 09 07:08:42.044330 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:42.044293    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:42.049342 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:42.049314    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:42.636186 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:42.636144    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:42.637169 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:42.637145    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:08:48.872742 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:48.872703    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:08:58.677265 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:58.677217    2562 patch_prober.go:28] interesting pod/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 07:08:58.677669 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:58.677285    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 07:08:59.738753 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:59.738663    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerID="c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2" exitCode=0
Sep 09 07:08:59.738753 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:59.738738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" event={"ID":"3ffcdecb-ae89-45b2-a783-b637189a43ca","Type":"ContainerDied","Data":"c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2"}
Sep 09 07:08:59.739285 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:59.739164    2562 scope.go:117] "RemoveContainer" containerID="c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2"
Sep 09 07:08:59.740255 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:59.740222    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:08:59.740345 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:59.740294    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:08:59.740345 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:08:59.740324    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:00.745679 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:00.745645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" event={"ID":"3ffcdecb-ae89-45b2-a783-b637189a43ca","Type":"ContainerStarted","Data":"a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3"}
Sep 09 07:09:01.043934 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:01.043836    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:09:01.044136 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:01.044087    2562 patch_prober.go:28] interesting pod/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 07:09:01.044228 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:01.044172    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 07:09:02.044147 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:02.044088    2562 patch_prober.go:28] interesting pod/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 07:09:02.044550 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:02.044169    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 07:09:02.912874 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:02.912836    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"]
Sep 09 07:09:02.913359 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:02.913338    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy"
Sep 09 07:09:02.913433 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:02.913362    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="004450e9-6a47-4f78-9b14-f355b707fd28" containerName="istio-proxy"
Sep 09 07:09:02.922987 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:02.922956    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:02.927434 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:02.927410    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-988b24-openshift-default-dockercfg-4hrr8\""
Sep 09 07:09:02.935441 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:02.935416    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"]
Sep 09 07:09:03.044776 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.044740    2562 patch_prober.go:28] interesting pod/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 07:09:03.045205 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.044805    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 07:09:03.087557 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.087512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-data\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.087835 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.087818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-envoy\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.087961 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.087946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-certs\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.088079 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.088066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-podinfo\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.088242 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.088227    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-socket\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.088389 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.088376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8818f8e1-f482-4a72-849c-4a94fb4f7207-istiod-ca-cert\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.088513 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.088499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-credential-socket\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.088643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.088630    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-token\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.088793 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.088779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn5gg\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-kube-api-access-kn5gg\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.189841 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.189793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8818f8e1-f482-4a72-849c-4a94fb4f7207-istiod-ca-cert\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.189856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-credential-socket\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.189899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-token\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.189955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn5gg\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-kube-api-access-kn5gg\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.189984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-data\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.190008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-envoy\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.190042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-certs\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.190069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-podinfo\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190516 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.190138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-socket\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.190509    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-socket\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190649 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.190626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8818f8e1-f482-4a72-849c-4a94fb4f7207-istiod-ca-cert\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190765 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.190737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-credential-socket\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.190881 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.190861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-data\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.194129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.192085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-certs\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.194129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.193568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-envoy\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.197505 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.197487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-podinfo\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.200377 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.200353    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-token\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.201282 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.201255    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn5gg\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-kube-api-access-kn5gg\") pod \"e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") " pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.233405 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.233373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:03.533207 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.533168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"]
Sep 09 07:09:03.534775 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:09:03.534751    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8818f8e1_f482_4a72_849c_4a94fb4f7207.slice/crio-268a0641c99a8f873b96d2f303bccab24f276d09b14d3768d9f64bc7779dedef WatchSource:0}: Error finding container 268a0641c99a8f873b96d2f303bccab24f276d09b14d3768d9f64bc7779dedef: Status 404 returned error can't find the container with id 268a0641c99a8f873b96d2f303bccab24f276d09b14d3768d9f64bc7779dedef
Sep 09 07:09:03.537317 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.537274    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:03.537398 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.537372    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:03.537446 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.537430    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:03.763428 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.763383    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" event={"ID":"8818f8e1-f482-4a72-849c-4a94fb4f7207","Type":"ContainerStarted","Data":"d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3"}
Sep 09 07:09:03.763625 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.763456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" event={"ID":"8818f8e1-f482-4a72-849c-4a94fb4f7207","Type":"ContainerStarted","Data":"268a0641c99a8f873b96d2f303bccab24f276d09b14d3768d9f64bc7779dedef"}
Sep 09 07:09:03.786380 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:03.786242    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podStartSLOduration=1.786219729 podStartE2EDuration="1.786219729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:09:03.784380161 +0000 UTC m=+1067.004621274" watchObservedRunningTime="2026-09-09 07:09:03.786219729 +0000 UTC m=+1067.006460843"
Sep 09 07:09:04.045123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:04.045023    2562 patch_prober.go:28] interesting pod/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 07:09:04.045123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:04.045077    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 07:09:04.234566 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:04.234527    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:04.239381 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:04.239358    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:04.773245 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:04.773206    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:04.778508 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:04.778479    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:05.044988 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:05.044903    2562 patch_prober.go:28] interesting pod/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 09 07:09:05.044988 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:05.044957    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 09 07:09:06.048372 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:06.048344    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:09:06.048775 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:06.048705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:09:06.049359 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:06.049339    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:09:12.568050 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:12.567960    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:09:15.858571 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:15.858539    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"]
Sep 09 07:09:15.860645 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:15.860630    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"
Sep 09 07:09:15.864296 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:15.864003    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:09:15.875136 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:15.874250    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:09:15.880068 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:15.880040    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"]
Sep 09 07:09:15.902123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:15.901867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911eaef9-f72a-4ccc-9e31-d2da276891d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c20824ba-99j6d\" (UID: \"911eaef9-f72a-4ccc-9e31-d2da276891d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"
Sep 09 07:09:16.002631 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:16.002591    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911eaef9-f72a-4ccc-9e31-d2da276891d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c20824ba-99j6d\" (UID: \"911eaef9-f72a-4ccc-9e31-d2da276891d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"
Sep 09 07:09:16.003248 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:16.003227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911eaef9-f72a-4ccc-9e31-d2da276891d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c20824ba-99j6d\" (UID: \"911eaef9-f72a-4ccc-9e31-d2da276891d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"
Sep 09 07:09:16.204126 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:16.204070    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"
Sep 09 07:09:16.478516 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:16.477630    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"]
Sep 09 07:09:16.480664 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:09:16.480611    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911eaef9_f72a_4ccc_9e31_d2da276891d2.slice/crio-d0595b3792c9315c239bb32eef0f884477ef606cf3dcd2fb053e04d941cc1538 WatchSource:0}: Error finding container d0595b3792c9315c239bb32eef0f884477ef606cf3dcd2fb053e04d941cc1538: Status 404 returned error can't find the container with id d0595b3792c9315c239bb32eef0f884477ef606cf3dcd2fb053e04d941cc1538
Sep 09 07:09:16.843785 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:16.843752    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d" event={"ID":"911eaef9-f72a-4ccc-9e31-d2da276891d2","Type":"ContainerStarted","Data":"6fe736a72bae83b11430ac52c900058b3c9a6b6863f16538e4c71b96b99b5a4a"}
Sep 09 07:09:16.843948 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:16.843817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d" event={"ID":"911eaef9-f72a-4ccc-9e31-d2da276891d2","Type":"ContainerStarted","Data":"d0595b3792c9315c239bb32eef0f884477ef606cf3dcd2fb053e04d941cc1538"}
Sep 09 07:09:16.860694 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:16.860648    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d" podStartSLOduration=1.8606288869999998 podStartE2EDuration="1.860628887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:09:16.860051061 +0000 UTC m=+1080.080292175" watchObservedRunningTime="2026-09-09 07:09:16.860628887 +0000 UTC m=+1080.080870001"
Sep 09 07:09:17.851024 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:17.850984    2562 generic.go:356] "Generic (PLEG): container finished" podID="911eaef9-f72a-4ccc-9e31-d2da276891d2" containerID="6fe736a72bae83b11430ac52c900058b3c9a6b6863f16538e4c71b96b99b5a4a" exitCode=0
Sep 09 07:09:17.851222 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:17.851066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d" event={"ID":"911eaef9-f72a-4ccc-9e31-d2da276891d2","Type":"ContainerDied","Data":"6fe736a72bae83b11430ac52c900058b3c9a6b6863f16538e4c71b96b99b5a4a"}
Sep 09 07:09:18.554543 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.554512    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"]
Sep 09 07:09:18.556679 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.556661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.559569 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.559543    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-43ead3\"/\"noten-model-43ead3-kserve-self-signed-certs\""
Sep 09 07:09:18.559679 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.559548    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-43ead3\"/\"kube-root-ca.crt\""
Sep 09 07:09:18.560505 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.560484    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-43ead3\"/\"default-dockercfg-xcwvg\""
Sep 09 07:09:18.560568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.560506    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-43ead3\"/\"openshift-service-ca.crt\""
Sep 09 07:09:18.573124 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.569921    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"]
Sep 09 07:09:18.622908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.622867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltfj7\" (UniqueName: \"kubernetes.io/projected/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-kube-api-access-ltfj7\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.623080 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.622926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-dshm\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.623080 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.622988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-home\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.623080 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.623056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-model-cache\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.623241 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.623088    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tmp-dir\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.623241 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.623130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.723534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.723498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-home\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.723736 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.723540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-model-cache\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.723736 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.723563    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tmp-dir\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.723736 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.723580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.723736 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.723607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltfj7\" (UniqueName: \"kubernetes.io/projected/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-kube-api-access-ltfj7\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.723736 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.723636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-dshm\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.723991 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.723965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-model-cache\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.724052 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.724011    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-home\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.724137 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.724054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tmp-dir\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.725937 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.725915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-dshm\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.726232 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.726216    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.732196 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.732170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltfj7\" (UniqueName: \"kubernetes.io/projected/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-kube-api-access-ltfj7\") pod \"noten-model-43ead3-kserve-6cf98dd675-xzf2v\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") " pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:18.876601 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:18.876332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:19.030873 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.030850    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"
Sep 09 07:09:19.126948 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.126862    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911eaef9-f72a-4ccc-9e31-d2da276891d2-maas-api-service-ca\") pod \"911eaef9-f72a-4ccc-9e31-d2da276891d2\" (UID: \"911eaef9-f72a-4ccc-9e31-d2da276891d2\") "
Sep 09 07:09:19.127287 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.127255    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911eaef9-f72a-4ccc-9e31-d2da276891d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "911eaef9-f72a-4ccc-9e31-d2da276891d2" (UID: "911eaef9-f72a-4ccc-9e31-d2da276891d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:09:19.185464 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.181408    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"]
Sep 09 07:09:19.227732 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.227706    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911eaef9-f72a-4ccc-9e31-d2da276891d2-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:19.863901 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.863792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v" event={"ID":"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de","Type":"ContainerStarted","Data":"3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac"}
Sep 09 07:09:19.864350 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.863921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v" event={"ID":"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de","Type":"ContainerStarted","Data":"1e930a2587246dbf1d44cbb9928c8ee306da863f77354e9000010f3b2676fa6c"}
Sep 09 07:09:19.865450 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.865426    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"
Sep 09 07:09:19.865578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.865448    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d" event={"ID":"911eaef9-f72a-4ccc-9e31-d2da276891d2","Type":"ContainerDied","Data":"d0595b3792c9315c239bb32eef0f884477ef606cf3dcd2fb053e04d941cc1538"}
Sep 09 07:09:19.865578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.865534    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0595b3792c9315c239bb32eef0f884477ef606cf3dcd2fb053e04d941cc1538"
Sep 09 07:09:19.882527 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:19.882484    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v" podStartSLOduration=1.882471728 podStartE2EDuration="1.882471728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:09:19.880383659 +0000 UTC m=+1083.100624786" watchObservedRunningTime="2026-09-09 07:09:19.882471728 +0000 UTC m=+1083.102712878"
Sep 09 07:09:20.702076 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:20.702043    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"]
Sep 09 07:09:20.775032 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:20.774961    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 07:09:20.775227 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:20.775076    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 07:09:20.942813 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:20.942769    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-self-signed-certs: secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:20.943342 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:20.942845    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs podName:15ad39a6-3a0e-4f30-8fb5-2313bf6b06de nodeName:}" failed. No retries permitted until 2026-09-09 07:09:21.442825272 +0000 UTC m=+1084.663066364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs") pod "noten-model-43ead3-kserve-6cf98dd675-xzf2v" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de") : secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:21.448244 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:21.448204    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-self-signed-certs: secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:21.448448 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:21.448267    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs podName:15ad39a6-3a0e-4f30-8fb5-2313bf6b06de nodeName:}" failed. No retries permitted until 2026-09-09 07:09:22.448254388 +0000 UTC m=+1085.668495479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs") pod "noten-model-43ead3-kserve-6cf98dd675-xzf2v" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de") : secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:22.456598 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:22.456561    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-self-signed-certs: secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:22.456973 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:22.456631    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs podName:15ad39a6-3a0e-4f30-8fb5-2313bf6b06de nodeName:}" failed. No retries permitted until 2026-09-09 07:09:24.45661698 +0000 UTC m=+1087.676858070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs") pod "noten-model-43ead3-kserve-6cf98dd675-xzf2v" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de") : secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:22.885643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:22.885527    2562 generic.go:356] "Generic (PLEG): container finished" podID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerID="d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3" exitCode=0
Sep 09 07:09:22.885643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:22.885612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" event={"ID":"8818f8e1-f482-4a72-849c-4a94fb4f7207","Type":"ContainerDied","Data":"d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3"}
Sep 09 07:09:22.886073 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:22.886057    2562 scope.go:117] "RemoveContainer" containerID="d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3"
Sep 09 07:09:22.887145 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:22.887118    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:22.887214 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:22.887189    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:22.887252 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:22.887225    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:23.900777 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:23.900743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" event={"ID":"8818f8e1-f482-4a72-849c-4a94fb4f7207","Type":"ContainerStarted","Data":"30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1"}
Sep 09 07:09:24.233543 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:24.233503    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:24.233847 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:24.233822    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 07:09:24.233905 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:24.233878    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 07:09:24.485076 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:24.484957    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-self-signed-certs: secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:24.485076 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:24.485063    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs podName:15ad39a6-3a0e-4f30-8fb5-2313bf6b06de nodeName:}" failed. No retries permitted until 2026-09-09 07:09:28.485042059 +0000 UTC m=+1091.705283153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs") pod "noten-model-43ead3-kserve-6cf98dd675-xzf2v" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de") : secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:25.234051 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:25.234017    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 07:09:25.234471 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:25.234067    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 07:09:26.234198 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:26.234161    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 07:09:26.234599 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:26.234226    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 07:09:26.532134 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:26.532017    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"]
Sep 09 07:09:26.534149 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:26.534126    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c20824ba-99j6d"]
Sep 09 07:09:27.231302 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:27.231271    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911eaef9-f72a-4ccc-9e31-d2da276891d2" path="/var/lib/kubelet/pods/911eaef9-f72a-4ccc-9e31-d2da276891d2/volumes"
Sep 09 07:09:27.234677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:27.234651    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 07:09:27.235025 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:27.234700    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 07:09:28.234081 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:28.234044    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 07:09:28.234297 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:28.234128    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 07:09:28.527989 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:28.527903    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-self-signed-certs: secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:28.527989 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:28.527981    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs podName:15ad39a6-3a0e-4f30-8fb5-2313bf6b06de nodeName:}" failed. No retries permitted until 2026-09-09 07:09:36.527962682 +0000 UTC m=+1099.748203782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs") pod "noten-model-43ead3-kserve-6cf98dd675-xzf2v" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de") : secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:28.582288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:28.582250    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:09:28.877683 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:28.877595    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:29.237522 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:29.237487    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:29.237764 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:29.237739    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:29.238305 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:29.238287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:09:32.937560 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:32.937526    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"]
Sep 09 07:09:32.937965 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:32.937770    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy" containerID="cri-o://a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3" gracePeriod=30
Sep 09 07:09:36.051535 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:36.051493    2562 patch_prober.go:28] interesting pod/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 07:09:36.051988 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:36.051556    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 07:09:36.606688 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:36.606652    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-self-signed-certs: secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:36.606874 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:36.606728    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs podName:15ad39a6-3a0e-4f30-8fb5-2313bf6b06de nodeName:}" failed. No retries permitted until 2026-09-09 07:09:52.60670942 +0000 UTC m=+1115.826950511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs") pod "noten-model-43ead3-kserve-6cf98dd675-xzf2v" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de") : secret "noten-model-43ead3-kserve-self-signed-certs" not found
Sep 09 07:09:36.898032 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:36.897942    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v" podUID="15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" containerName="main" containerID="cri-o://3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac" gracePeriod=45
Sep 09 07:09:37.176253 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.176228    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:37.313475 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313440    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs\") pod \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") "
Sep 09 07:09:37.313671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313504    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tmp-dir\") pod \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") "
Sep 09 07:09:37.313671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313529    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-dshm\") pod \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") "
Sep 09 07:09:37.313671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313571    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltfj7\" (UniqueName: \"kubernetes.io/projected/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-kube-api-access-ltfj7\") pod \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") "
Sep 09 07:09:37.313671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313602    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-model-cache\") pod \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") "
Sep 09 07:09:37.313671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313625    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-home\") pod \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\" (UID: \"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de\") "
Sep 09 07:09:37.313967 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313791    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:37.313967 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313904    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-model-cache" (OuterVolumeSpecName: "model-cache") pod "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:37.313967 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313938    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:37.314080 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.313997    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-home" (OuterVolumeSpecName: "home") pod "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:37.315853 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.315825    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-dshm" (OuterVolumeSpecName: "dshm") pod "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:37.315853 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.315833    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:09:37.316007 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.315890    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-kube-api-access-ltfj7" (OuterVolumeSpecName: "kube-api-access-ltfj7") pod "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" (UID: "15ad39a6-3a0e-4f30-8fb5-2313bf6b06de"). InnerVolumeSpecName "kube-api-access-ltfj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:09:37.414627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.414543    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:37.414627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.414570    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltfj7\" (UniqueName: \"kubernetes.io/projected/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-kube-api-access-ltfj7\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:37.414627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.414580    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:37.414627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.414590    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:37.414627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.414597    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:37.985381 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.985349    2562 generic.go:356] "Generic (PLEG): container finished" podID="15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" containerID="3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac" exitCode=0
Sep 09 07:09:37.985528 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.985391    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"
Sep 09 07:09:37.985528 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.985420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v" event={"ID":"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de","Type":"ContainerDied","Data":"3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac"}
Sep 09 07:09:37.985528 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.985461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v" event={"ID":"15ad39a6-3a0e-4f30-8fb5-2313bf6b06de","Type":"ContainerDied","Data":"1e930a2587246dbf1d44cbb9928c8ee306da863f77354e9000010f3b2676fa6c"}
Sep 09 07:09:37.985528 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:37.985480    2562 scope.go:117] "RemoveContainer" containerID="3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac"
Sep 09 07:09:38.023931 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.023896    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"]
Sep 09 07:09:38.028777 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.028754    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-43ead3/noten-model-43ead3-kserve-6cf98dd675-xzf2v"]
Sep 09 07:09:38.050424 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.050398    2562 scope.go:117] "RemoveContainer" containerID="3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac"
Sep 09 07:09:38.050789 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:38.050769    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac\": container with ID starting with 3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac not found: ID does not exist" containerID="3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac"
Sep 09 07:09:38.050868 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.050803    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac"} err="failed to get container status \"3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac\": rpc error: code = NotFound desc = could not find container \"3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac\": container with ID starting with 3a99bf82e1782ff5c76b37d0731b93018edef26de99227944648786942ce68ac not found: ID does not exist"
Sep 09 07:09:38.198520 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.198492    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:09:38.321387 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321302    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvj7l\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-kube-api-access-hvj7l\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.321387 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321347    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-credential-socket\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.321387 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321379    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-certs\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.321671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321437    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-envoy\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.321671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321485    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-socket\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.321671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321510    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-data\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.321671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321538    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-token\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.321671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321559    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:38.321671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321577    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffcdecb-ae89-45b2-a783-b637189a43ca-istiod-ca-cert\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.321671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321649    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-podinfo\") pod \"3ffcdecb-ae89-45b2-a783-b637189a43ca\" (UID: \"3ffcdecb-ae89-45b2-a783-b637189a43ca\") "
Sep 09 07:09:38.322047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321760    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:38.322047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321774    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:38.322047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321786    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:38.322047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321982    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.322047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.322002    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.322047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.321999    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffcdecb-ae89-45b2-a783-b637189a43ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:09:38.322047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.322016    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.322047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.322029    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.323612 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.323585    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-kube-api-access-hvj7l" (OuterVolumeSpecName: "kube-api-access-hvj7l") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "kube-api-access-hvj7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:09:38.323758 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.323735    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:09:38.323814 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.323744    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:09:38.323853 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.323832    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ffcdecb-ae89-45b2-a783-b637189a43ca" (UID: "3ffcdecb-ae89-45b2-a783-b637189a43ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:09:38.422593 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.422556    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.422593 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.422589    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvj7l\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-kube-api-access-hvj7l\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.422593 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.422599    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.422593 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.422607    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffcdecb-ae89-45b2-a783-b637189a43ca-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.422811 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.422615    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffcdecb-ae89-45b2-a783-b637189a43ca-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:09:38.993682 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.993649    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerID="a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3" exitCode=0
Sep 09 07:09:38.993883 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.993705    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"
Sep 09 07:09:38.993883 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.993736    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" event={"ID":"3ffcdecb-ae89-45b2-a783-b637189a43ca","Type":"ContainerDied","Data":"a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3"}
Sep 09 07:09:38.993883 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.993794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx" event={"ID":"3ffcdecb-ae89-45b2-a783-b637189a43ca","Type":"ContainerDied","Data":"63ac3bbe4f43aa7e1d7327287c910398b5d4231c17be1619cbb34a65f86907b2"}
Sep 09 07:09:38.993883 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:38.993819    2562 scope.go:117] "RemoveContainer" containerID="a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3"
Sep 09 07:09:39.003492 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.003473    2562 scope.go:117] "RemoveContainer" containerID="c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2"
Sep 09 07:09:39.011689 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.011673    2562 scope.go:117] "RemoveContainer" containerID="a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3"
Sep 09 07:09:39.011947 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:39.011926    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3\": container with ID starting with a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3 not found: ID does not exist" containerID="a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3"
Sep 09 07:09:39.011993 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.011955    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3"} err="failed to get container status \"a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3\": rpc error: code = NotFound desc = could not find container \"a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3\": container with ID starting with a23d9ef041b571dab51904f29f213cb0631b14e9c523436bd02c44d5d4de60a3 not found: ID does not exist"
Sep 09 07:09:39.011993 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.011972    2562 scope.go:117] "RemoveContainer" containerID="c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2"
Sep 09 07:09:39.012232 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:09:39.012216    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2\": container with ID starting with c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2 not found: ID does not exist" containerID="c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2"
Sep 09 07:09:39.012283 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.012236    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2"} err="failed to get container status \"c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2\": rpc error: code = NotFound desc = could not find container \"c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2\": container with ID starting with c9d4c70787926e6a701022952a5475b191539502ba6281715a08115ccc8929a2 not found: ID does not exist"
Sep 09 07:09:39.022363 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.022337    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"]
Sep 09 07:09:39.027258 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.027230    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c20824ba-openshift-default-7755448bc6-kpswx"]
Sep 09 07:09:39.229672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.229637    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" path="/var/lib/kubelet/pods/15ad39a6-3a0e-4f30-8fb5-2313bf6b06de/volumes"
Sep 09 07:09:39.230049 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:39.230035    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" path="/var/lib/kubelet/pods/3ffcdecb-ae89-45b2-a783-b637189a43ca/volumes"
Sep 09 07:09:40.315769 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:40.315732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:09:47.329927 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.329887    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"]
Sep 09 07:09:47.330373 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330357    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" containerName="main"
Sep 09 07:09:47.330421 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330375    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" containerName="main"
Sep 09 07:09:47.330421 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330399    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911eaef9-f72a-4ccc-9e31-d2da276891d2" containerName="revoke-keys"
Sep 09 07:09:47.330421 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330409    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="911eaef9-f72a-4ccc-9e31-d2da276891d2" containerName="revoke-keys"
Sep 09 07:09:47.330521 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330426    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy"
Sep 09 07:09:47.330521 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330435    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy"
Sep 09 07:09:47.330587 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330542    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="911eaef9-f72a-4ccc-9e31-d2da276891d2" containerName="revoke-keys"
Sep 09 07:09:47.330587 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330553    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ad39a6-3a0e-4f30-8fb5-2313bf6b06de" containerName="main"
Sep 09 07:09:47.330587 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330563    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy"
Sep 09 07:09:47.330587 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.330574    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy"
Sep 09 07:09:47.332804 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.332785    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.335416 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.335395    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-1128ac-openshift-default-dockercfg-cr6q6\""
Sep 09 07:09:47.346664 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.346638    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"]
Sep 09 07:09:47.499623 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-envoy\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.499811 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499692    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-credential-socket\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.499811 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-token\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.499811 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-certs\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.499964 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-podinfo\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.499964 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwq5r\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-kube-api-access-dwq5r\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.499964 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-data\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.499964 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istiod-ca-cert\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.500207 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.499991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-socket\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.600715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.600627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-credential-socket\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.600715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.600686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-token\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.600947 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.600730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-certs\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.600947 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.600782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-podinfo\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.600947 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.600812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwq5r\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-kube-api-access-dwq5r\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.600947 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.600844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-data\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.600947 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.600864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istiod-ca-cert\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.600947 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.600892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-socket\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.601290 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.601081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-credential-socket\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.601290 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.601129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-certs\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.601393 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.601287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-envoy\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.601393 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.601332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-socket\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.601393 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.601287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-data\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.601612 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.601590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istiod-ca-cert\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.603212 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.603171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-envoy\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.603447 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.603429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-podinfo\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.609119 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.609082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-token\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.609204 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.609165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwq5r\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-kube-api-access-dwq5r\") pod \"e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") " pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.643729 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.643710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:47.928879 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.928852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"]
Sep 09 07:09:47.929951 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:09:47.929921    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb91a24_9fab_4fc7_a613_2a901881c5dc.slice/crio-22c52070c7bf151667f26b844f6893323d316b5b943167dcd0b1b6136059007d WatchSource:0}: Error finding container 22c52070c7bf151667f26b844f6893323d316b5b943167dcd0b1b6136059007d: Status 404 returned error can't find the container with id 22c52070c7bf151667f26b844f6893323d316b5b943167dcd0b1b6136059007d
Sep 09 07:09:47.932137 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.932087    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:47.932220 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.932165    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:47.932220 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:47.932195    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:09:48.044838 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:48.044805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" event={"ID":"7fb91a24-9fab-4fc7-a613-2a901881c5dc","Type":"ContainerStarted","Data":"5cfd3ab1f49c92415dd370eeb745555e049f970ae1fb7b57d2b0cf844355a7b2"}
Sep 09 07:09:48.045016 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:48.044869    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" event={"ID":"7fb91a24-9fab-4fc7-a613-2a901881c5dc","Type":"ContainerStarted","Data":"22c52070c7bf151667f26b844f6893323d316b5b943167dcd0b1b6136059007d"}
Sep 09 07:09:48.644955 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:48.644920    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:48.649659 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:48.649637    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:48.673697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:48.673652    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podStartSLOduration=1.6736376979999998 podStartE2EDuration="1.673637698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:09:48.066324575 +0000 UTC m=+1111.286565691" watchObservedRunningTime="2026-09-09 07:09:48.673637698 +0000 UTC m=+1111.893878811"
Sep 09 07:09:49.050423 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:49.050390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:49.051256 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:49.051240    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:09:56.967760 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:09:56.967722    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:10:05.091263 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:05.091222    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 07:10:05.091656 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:05.091298    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 07:10:07.157799 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:07.157763    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerID="5cfd3ab1f49c92415dd370eeb745555e049f970ae1fb7b57d2b0cf844355a7b2" exitCode=0
Sep 09 07:10:07.158291 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:07.157837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" event={"ID":"7fb91a24-9fab-4fc7-a613-2a901881c5dc","Type":"ContainerDied","Data":"5cfd3ab1f49c92415dd370eeb745555e049f970ae1fb7b57d2b0cf844355a7b2"}
Sep 09 07:10:07.158291 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:07.158263    2562 scope.go:117] "RemoveContainer" containerID="5cfd3ab1f49c92415dd370eeb745555e049f970ae1fb7b57d2b0cf844355a7b2"
Sep 09 07:10:07.159307 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:07.159277    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:07.159369 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:07.159347    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:07.159405 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:07.159379    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:08.165772 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:08.165734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" event={"ID":"7fb91a24-9fab-4fc7-a613-2a901881c5dc","Type":"ContainerStarted","Data":"6040f94d8c76038a9786e296c258f235cac8b0c44fa5bba6f8cc0ea53831aa7c"}
Sep 09 07:10:08.644878 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:08.644788    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:10:08.645033 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:08.644978    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 09 07:10:08.645080 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:08.645034    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 09 07:10:09.644678 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:09.644639    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 09 07:10:09.645064 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:09.644698    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 09 07:10:10.644246 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:10.644206    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 09 07:10:10.644428 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:10.644291    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 09 07:10:11.359249 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.359213    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"]
Sep 09 07:10:11.359604 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.359549    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy"
Sep 09 07:10:11.359604 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.359560    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffcdecb-ae89-45b2-a783-b637189a43ca" containerName="istio-proxy"
Sep 09 07:10:11.361469 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.361450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.375221 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.375186    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"]
Sep 09 07:10:11.375471 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.375384    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-988b24\"/\"openshift-service-ca.crt\""
Sep 09 07:10:11.375672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.375624    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-988b24\"/\"kube-root-ca.crt\""
Sep 09 07:10:11.375871 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.375847    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-988b24\"/\"auth-test-model-988b24-kserve-self-signed-certs\""
Sep 09 07:10:11.376112 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.376079    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-988b24\"/\"default-dockercfg-7bjfk\""
Sep 09 07:10:11.414201 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.414167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdjm\" (UniqueName: \"kubernetes.io/projected/65aca61e-a829-4c8f-94ed-b27fd26da039-kube-api-access-6bdjm\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.414361 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.414208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-model-cache\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.414361 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.414239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-dshm\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.414361 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.414319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-tmp-dir\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.414491 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.414377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65aca61e-a829-4c8f-94ed-b27fd26da039-tls-certs\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.414491 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.414435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-home\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515304 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65aca61e-a829-4c8f-94ed-b27fd26da039-tls-certs\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-home\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdjm\" (UniqueName: \"kubernetes.io/projected/65aca61e-a829-4c8f-94ed-b27fd26da039-kube-api-access-6bdjm\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-model-cache\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-dshm\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515703 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-tmp-dir\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515859 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515831    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-model-cache\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515928 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-home\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.515928 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.515912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-tmp-dir\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.517753 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.517726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-dshm\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.517901 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.517884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65aca61e-a829-4c8f-94ed-b27fd26da039-tls-certs\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.524567 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.524542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdjm\" (UniqueName: \"kubernetes.io/projected/65aca61e-a829-4c8f-94ed-b27fd26da039-kube-api-access-6bdjm\") pod \"auth-test-model-988b24-kserve-64f4864659-dtczf\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") " pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.645014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.644922    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 09 07:10:11.645014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.644982    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 09 07:10:11.684508 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.684471    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:11.959454 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:11.959428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"]
Sep 09 07:10:11.961247 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:10:11.961219    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65aca61e_a829_4c8f_94ed_b27fd26da039.slice/crio-cb8d14b37c80b77a62614e9feebb193d16d7fbddfaba3efee2c2ddd95a7755cc WatchSource:0}: Error finding container cb8d14b37c80b77a62614e9feebb193d16d7fbddfaba3efee2c2ddd95a7755cc: Status 404 returned error can't find the container with id cb8d14b37c80b77a62614e9feebb193d16d7fbddfaba3efee2c2ddd95a7755cc
Sep 09 07:10:12.200456 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:12.200419    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf" event={"ID":"65aca61e-a829-4c8f-94ed-b27fd26da039","Type":"ContainerStarted","Data":"d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81"}
Sep 09 07:10:12.200645 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:12.200472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf" event={"ID":"65aca61e-a829-4c8f-94ed-b27fd26da039","Type":"ContainerStarted","Data":"cb8d14b37c80b77a62614e9feebb193d16d7fbddfaba3efee2c2ddd95a7755cc"}
Sep 09 07:10:12.224520 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:12.224433    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf" podStartSLOduration=1.224416293 podStartE2EDuration="1.224416293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:10:12.223703113 +0000 UTC m=+1135.443944226" watchObservedRunningTime="2026-09-09 07:10:12.224416293 +0000 UTC m=+1135.444657408"
Sep 09 07:10:12.644318 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:12.644224    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 09 07:10:12.644318 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:12.644285    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 09 07:10:13.648401 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:13.648366    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:10:13.648871 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:13.648676    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:10:13.649325 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:13.649309    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:10:15.371893 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.371857    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:10:15.671855 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.671822    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"]
Sep 09 07:10:15.674454 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.674430    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.677060 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.677036    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-1d8843-openshift-default-dockercfg-wb7db\""
Sep 09 07:10:15.694038 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.694008    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"]
Sep 09 07:10:15.754322 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754288    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732324b1-73bf-4d7c-9317-648b663c5b22-istio-podinfo\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.754521 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754328    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-data\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.754521 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2jj5\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-kube-api-access-s2jj5\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.754521 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-socket\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.754675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-credential-socket\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.754675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-envoy\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.754675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732324b1-73bf-4d7c-9317-648b663c5b22-istiod-ca-cert\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.754675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-certs\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.754820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.754706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-istio-token\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.855983 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.855946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-socket\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-credential-socket\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-envoy\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856206 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732324b1-73bf-4d7c-9317-648b663c5b22-istiod-ca-cert\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856394 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-certs\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856456 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-istio-token\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856456 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-credential-socket\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856459    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732324b1-73bf-4d7c-9317-648b663c5b22-istio-podinfo\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-socket\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-data\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856527    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2jj5\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-kube-api-access-s2jj5\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856776 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-certs\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.856859 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.856823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-data\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.857422 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.857390    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732324b1-73bf-4d7c-9317-648b663c5b22-istiod-ca-cert\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.858669 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.858649    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-envoy\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.859059 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.859037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732324b1-73bf-4d7c-9317-648b663c5b22-istio-podinfo\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.864634 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.864593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-istio-token\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.864992 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.864973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2jj5\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-kube-api-access-s2jj5\") pod \"e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") " pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:15.986370 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:15.986286    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:16.290328 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:16.290296    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"]
Sep 09 07:10:16.291477 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:10:16.291441    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732324b1_73bf_4d7c_9317_648b663c5b22.slice/crio-61f00ee4074ab33171e27e7e5c3ceeb282a9217199230ded07d40e855e43b7e1 WatchSource:0}: Error finding container 61f00ee4074ab33171e27e7e5c3ceeb282a9217199230ded07d40e855e43b7e1: Status 404 returned error can't find the container with id 61f00ee4074ab33171e27e7e5c3ceeb282a9217199230ded07d40e855e43b7e1
Sep 09 07:10:16.294650 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:16.294609    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:16.294746 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:16.294687    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:16.294746 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:16.294726    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:17.238271 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:17.238239    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" event={"ID":"732324b1-73bf-4d7c-9317-648b663c5b22","Type":"ContainerStarted","Data":"8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319"}
Sep 09 07:10:17.238638 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:17.238284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" event={"ID":"732324b1-73bf-4d7c-9317-648b663c5b22","Type":"ContainerStarted","Data":"61f00ee4074ab33171e27e7e5c3ceeb282a9217199230ded07d40e855e43b7e1"}
Sep 09 07:10:17.261709 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:17.261660    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" podStartSLOduration=2.261644269 podStartE2EDuration="2.261644269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:10:17.260034033 +0000 UTC m=+1140.480275146" watchObservedRunningTime="2026-09-09 07:10:17.261644269 +0000 UTC m=+1140.481885775"
Sep 09 07:10:17.292527 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:17.292498    2562 scope.go:117] "RemoveContainer" containerID="aec8f948f80fbe9a271d1780f5e2ac3e3ad7591f92616094dfc072e116fb5e71"
Sep 09 07:10:17.986597 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:17.986569    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:17.991231 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:17.991208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:18.244314 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:18.244226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:18.245180 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:18.245162    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:21.059226 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.059195    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"]
Sep 09 07:10:21.062291 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.062272    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:21.064773 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.064750    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-1d8843\""
Sep 09 07:10:21.064900 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.064758    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-1d8843-dockercfg-jbbq2\""
Sep 09 07:10:21.074044 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.074023    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"]
Sep 09 07:10:21.104006 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.103975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5\" (UID: \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:21.104261 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.104237    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqsd\" (UniqueName: \"kubernetes.io/projected/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-kube-api-access-sfqsd\") pod \"payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5\" (UID: \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:21.205296 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.205258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqsd\" (UniqueName: \"kubernetes.io/projected/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-kube-api-access-sfqsd\") pod \"payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5\" (UID: \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:21.205496 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.205379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5\" (UID: \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:21.206764 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.206730    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5\" (UID: \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:21.214877 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.214844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqsd\" (UniqueName: \"kubernetes.io/projected/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-kube-api-access-sfqsd\") pod \"payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5\" (UID: \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:21.374468 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.374374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:21.649455 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.649421    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"]
Sep 09 07:10:21.650813 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:10:21.650785    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e01c5d7_4456_41d8_a9f1_cf8cd40cdcbd.slice/crio-266d9e5d718815053e20533f97a5ad7acc45ae52791af12adeeac69e26b48747 WatchSource:0}: Error finding container 266d9e5d718815053e20533f97a5ad7acc45ae52791af12adeeac69e26b48747: Status 404 returned error can't find the container with id 266d9e5d718815053e20533f97a5ad7acc45ae52791af12adeeac69e26b48747
Sep 09 07:10:21.652858 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.652842    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 07:10:21.685266 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.685240    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:21.685379 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.685291    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:21.697676 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:21.697652    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:22.275434 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:22.275328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5" event={"ID":"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd","Type":"ContainerStarted","Data":"266d9e5d718815053e20533f97a5ad7acc45ae52791af12adeeac69e26b48747"}
Sep 09 07:10:22.290872 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:22.290832    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:10:23.283186 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:23.283144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5" event={"ID":"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd","Type":"ContainerStarted","Data":"b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555"}
Sep 09 07:10:23.304414 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:23.304358    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5" podStartSLOduration=0.802589231 podStartE2EDuration="2.304342279s" totalImagesPullingTime="1.501753048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:10:23.302245502 +0000 UTC m=+1146.522486616" watchObservedRunningTime="2026-09-09 07:10:23.304342279 +0000 UTC m=+1146.524583433"
Sep 09 07:10:24.288397 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:24.288354    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:28.277403 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:28.277363    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:10:35.295294 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:35.295257    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:10:38.371752 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:38.371719    2562 generic.go:356] "Generic (PLEG): container finished" podID="732324b1-73bf-4d7c-9317-648b663c5b22" containerID="8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319" exitCode=0
Sep 09 07:10:38.372245 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:38.371797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" event={"ID":"732324b1-73bf-4d7c-9317-648b663c5b22","Type":"ContainerDied","Data":"8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319"}
Sep 09 07:10:38.372338 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:38.372261    2562 scope.go:117] "RemoveContainer" containerID="8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319"
Sep 09 07:10:38.373470 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:38.373439    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:38.373590 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:38.373521    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:38.373590 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:38.373571    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:10:39.384017 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:39.383976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" event={"ID":"732324b1-73bf-4d7c-9317-648b663c5b22","Type":"ContainerStarted","Data":"8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1"}
Sep 09 07:10:39.768658 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:39.768619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:10:39.986538 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:39.986448    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:39.986774 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:39.986743    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 07:10:39.986859 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:39.986812    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 07:10:40.987684 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:40.987642    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 07:10:40.988342 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:40.988297    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 07:10:41.986827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:41.986792    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 07:10:41.987136 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:41.987092    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 07:10:42.987438 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:42.987391    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 07:10:42.987964 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:42.987461    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 07:10:43.987788 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:43.987743    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 09 07:10:43.988437 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:43.988388    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 09 07:10:44.901454 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:44.901418    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:44.904500 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:44.904483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:44.907065 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:44.907048    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dw24h\""
Sep 09 07:10:44.911029 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:44.910997    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:44.991397 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:44.991368    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:44.991827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:44.991809    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:44.992532 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:44.992509    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:10:45.038126 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:45.038074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkljm\" (UniqueName: \"kubernetes.io/projected/530c2d1b-c5bb-42d5-8711-b581ad23822d-kube-api-access-tkljm\") pod \"test-curl-7943-140414780036992\" (UID: \"530c2d1b-c5bb-42d5-8711-b581ad23822d\") " pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:45.139112 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:45.139075    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkljm\" (UniqueName: \"kubernetes.io/projected/530c2d1b-c5bb-42d5-8711-b581ad23822d-kube-api-access-tkljm\") pod \"test-curl-7943-140414780036992\" (UID: \"530c2d1b-c5bb-42d5-8711-b581ad23822d\") " pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:45.147445 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:45.147422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkljm\" (UniqueName: \"kubernetes.io/projected/530c2d1b-c5bb-42d5-8711-b581ad23822d-kube-api-access-tkljm\") pod \"test-curl-7943-140414780036992\" (UID: \"530c2d1b-c5bb-42d5-8711-b581ad23822d\") " pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:45.215168 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:45.215143    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:45.488589 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:45.488549    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:45.491123 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:10:45.491071    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod530c2d1b_c5bb_42d5_8711_b581ad23822d.slice/crio-79a6fcfeff8be5bbbb52f6cd42f67a604fb282178c2775d050448cd274d2a2db WatchSource:0}: Error finding container 79a6fcfeff8be5bbbb52f6cd42f67a604fb282178c2775d050448cd274d2a2db: Status 404 returned error can't find the container with id 79a6fcfeff8be5bbbb52f6cd42f67a604fb282178c2775d050448cd274d2a2db
Sep 09 07:10:46.430666 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:46.430629    2562 generic.go:356] "Generic (PLEG): container finished" podID="530c2d1b-c5bb-42d5-8711-b581ad23822d" containerID="e09db6d46f6025d8c77bf874da4d3f6033172fc699c95a6f8f594d8fab44a5ed" exitCode=0
Sep 09 07:10:46.431053 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:46.430673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-140414780036992" event={"ID":"530c2d1b-c5bb-42d5-8711-b581ad23822d","Type":"ContainerDied","Data":"e09db6d46f6025d8c77bf874da4d3f6033172fc699c95a6f8f594d8fab44a5ed"}
Sep 09 07:10:46.431053 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:46.430726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-140414780036992" event={"ID":"530c2d1b-c5bb-42d5-8711-b581ad23822d","Type":"ContainerStarted","Data":"79a6fcfeff8be5bbbb52f6cd42f67a604fb282178c2775d050448cd274d2a2db"}
Sep 09 07:10:47.563585 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.563559    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:47.600870 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.600845    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7943-140414780036992_530c2d1b-c5bb-42d5-8711-b581ad23822d/test-curl-7943-140414780036992/0.log"
Sep 09 07:10:47.625516 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.625490    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:47.631165 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.631137    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:47.663214 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.663192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkljm\" (UniqueName: \"kubernetes.io/projected/530c2d1b-c5bb-42d5-8711-b581ad23822d-kube-api-access-tkljm\") pod \"530c2d1b-c5bb-42d5-8711-b581ad23822d\" (UID: \"530c2d1b-c5bb-42d5-8711-b581ad23822d\") "
Sep 09 07:10:47.665226 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.665202    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530c2d1b-c5bb-42d5-8711-b581ad23822d-kube-api-access-tkljm" (OuterVolumeSpecName: "kube-api-access-tkljm") pod "530c2d1b-c5bb-42d5-8711-b581ad23822d" (UID: "530c2d1b-c5bb-42d5-8711-b581ad23822d"). InnerVolumeSpecName "kube-api-access-tkljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:10:47.750035 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.749967    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:47.750363 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.750349    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="530c2d1b-c5bb-42d5-8711-b581ad23822d" containerName="test-curl-7943-140414780036992"
Sep 09 07:10:47.750405 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.750367    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="530c2d1b-c5bb-42d5-8711-b581ad23822d" containerName="test-curl-7943-140414780036992"
Sep 09 07:10:47.750444 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.750435    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="530c2d1b-c5bb-42d5-8711-b581ad23822d" containerName="test-curl-7943-140414780036992"
Sep 09 07:10:47.752305 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.752290    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:47.763058 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.763032    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:47.764379 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.764354    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkljm\" (UniqueName: \"kubernetes.io/projected/530c2d1b-c5bb-42d5-8711-b581ad23822d-kube-api-access-tkljm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:10:47.865719 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.865679    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbbj\" (UniqueName: \"kubernetes.io/projected/b6131e94-d073-4650-b147-59508e676632-kube-api-access-prbbj\") pod \"test-curl-7943-140414780036992\" (UID: \"b6131e94-d073-4650-b147-59508e676632\") " pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:47.966826 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.966794    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbbj\" (UniqueName: \"kubernetes.io/projected/b6131e94-d073-4650-b147-59508e676632-kube-api-access-prbbj\") pod \"test-curl-7943-140414780036992\" (UID: \"b6131e94-d073-4650-b147-59508e676632\") " pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:47.975146 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:47.975122    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbbj\" (UniqueName: \"kubernetes.io/projected/b6131e94-d073-4650-b147-59508e676632-kube-api-access-prbbj\") pod \"test-curl-7943-140414780036992\" (UID: \"b6131e94-d073-4650-b147-59508e676632\") " pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:48.061646 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:48.061567    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:48.330257 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:48.330181    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:48.333294 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:10:48.333267    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6131e94_d073_4650_b147_59508e676632.slice/crio-a50a2802139ff9b0869fc2e46b4133127ff9a07e32fe5edec87b2964ba89efa7 WatchSource:0}: Error finding container a50a2802139ff9b0869fc2e46b4133127ff9a07e32fe5edec87b2964ba89efa7: Status 404 returned error can't find the container with id a50a2802139ff9b0869fc2e46b4133127ff9a07e32fe5edec87b2964ba89efa7
Sep 09 07:10:48.442846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:48.442816    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-140414780036992" event={"ID":"b6131e94-d073-4650-b147-59508e676632","Type":"ContainerStarted","Data":"a50a2802139ff9b0869fc2e46b4133127ff9a07e32fe5edec87b2964ba89efa7"}
Sep 09 07:10:48.444128 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:48.444088    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:48.444249 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:48.444088    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79a6fcfeff8be5bbbb52f6cd42f67a604fb282178c2775d050448cd274d2a2db"
Sep 09 07:10:48.456636 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:48.456605    2562 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7943-140414780036992" oldPodUID="530c2d1b-c5bb-42d5-8711-b581ad23822d" podUID="b6131e94-d073-4650-b147-59508e676632"
Sep 09 07:10:49.230319 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:49.230293    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="530c2d1b-c5bb-42d5-8711-b581ad23822d" path="/var/lib/kubelet/pods/530c2d1b-c5bb-42d5-8711-b581ad23822d/volumes"
Sep 09 07:10:49.451887 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:49.451852    2562 generic.go:356] "Generic (PLEG): container finished" podID="b6131e94-d073-4650-b147-59508e676632" containerID="4e1896af4be1270601199007a8669bcb532eda3336a61ca138117a1296715864" exitCode=0
Sep 09 07:10:49.452062 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:49.451941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-140414780036992" event={"ID":"b6131e94-d073-4650-b147-59508e676632","Type":"ContainerDied","Data":"4e1896af4be1270601199007a8669bcb532eda3336a61ca138117a1296715864"}
Sep 09 07:10:50.588346 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.588326    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:50.588644 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.588631    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prbbj\" (UniqueName: \"kubernetes.io/projected/b6131e94-d073-4650-b147-59508e676632-kube-api-access-prbbj\") pod \"b6131e94-d073-4650-b147-59508e676632\" (UID: \"b6131e94-d073-4650-b147-59508e676632\") "
Sep 09 07:10:50.590534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.590510    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6131e94-d073-4650-b147-59508e676632-kube-api-access-prbbj" (OuterVolumeSpecName: "kube-api-access-prbbj") pod "b6131e94-d073-4650-b147-59508e676632" (UID: "b6131e94-d073-4650-b147-59508e676632"). InnerVolumeSpecName "kube-api-access-prbbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:10:50.606867 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.606843    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7943-140414780036992_b6131e94-d073-4650-b147-59508e676632/test-curl-7943-140414780036992/0.log"
Sep 09 07:10:50.627158 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.627124    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:50.631174 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.631151    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7943-140414780036992"]
Sep 09 07:10:50.689385 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.689352    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prbbj\" (UniqueName: \"kubernetes.io/projected/b6131e94-d073-4650-b147-59508e676632-kube-api-access-prbbj\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:10:50.770557 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.770487    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:50.770854 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.770840    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6131e94-d073-4650-b147-59508e676632" containerName="test-curl-7943-140414780036992"
Sep 09 07:10:50.770892 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.770857    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6131e94-d073-4650-b147-59508e676632" containerName="test-curl-7943-140414780036992"
Sep 09 07:10:50.770940 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.770931    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6131e94-d073-4650-b147-59508e676632" containerName="test-curl-7943-140414780036992"
Sep 09 07:10:50.773078 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.773063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:50.787526 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.787489    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:50.790146 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.790122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74pk\" (UniqueName: \"kubernetes.io/projected/ecda6810-8fca-4c32-a3df-2fe5cb7d5282-kube-api-access-r74pk\") pod \"test-curl-7943-140414780037568\" (UID: \"ecda6810-8fca-4c32-a3df-2fe5cb7d5282\") " pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:50.891512 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.891477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74pk\" (UniqueName: \"kubernetes.io/projected/ecda6810-8fca-4c32-a3df-2fe5cb7d5282-kube-api-access-r74pk\") pod \"test-curl-7943-140414780037568\" (UID: \"ecda6810-8fca-4c32-a3df-2fe5cb7d5282\") " pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:50.900488 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:50.900464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74pk\" (UniqueName: \"kubernetes.io/projected/ecda6810-8fca-4c32-a3df-2fe5cb7d5282-kube-api-access-r74pk\") pod \"test-curl-7943-140414780037568\" (UID: \"ecda6810-8fca-4c32-a3df-2fe5cb7d5282\") " pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:51.090148 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:51.090053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:51.229712 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:51.229678    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6131e94-d073-4650-b147-59508e676632" path="/var/lib/kubelet/pods/b6131e94-d073-4650-b147-59508e676632/volumes"
Sep 09 07:10:51.360987 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:51.360961    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:51.363366 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:10:51.363338    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecda6810_8fca_4c32_a3df_2fe5cb7d5282.slice/crio-69c9d6b3495c561226f200ed5ab8f2d5c583af6671b87d7f7d5468c0d727d6fd WatchSource:0}: Error finding container 69c9d6b3495c561226f200ed5ab8f2d5c583af6671b87d7f7d5468c0d727d6fd: Status 404 returned error can't find the container with id 69c9d6b3495c561226f200ed5ab8f2d5c583af6671b87d7f7d5468c0d727d6fd
Sep 09 07:10:51.465000 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:51.464964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-140414780037568" event={"ID":"ecda6810-8fca-4c32-a3df-2fe5cb7d5282","Type":"ContainerStarted","Data":"69c9d6b3495c561226f200ed5ab8f2d5c583af6671b87d7f7d5468c0d727d6fd"}
Sep 09 07:10:51.466297 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:51.466273    2562 scope.go:117] "RemoveContainer" containerID="4e1896af4be1270601199007a8669bcb532eda3336a61ca138117a1296715864"
Sep 09 07:10:51.466414 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:51.466279    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780036992"
Sep 09 07:10:52.472973 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:52.472889    2562 generic.go:356] "Generic (PLEG): container finished" podID="ecda6810-8fca-4c32-a3df-2fe5cb7d5282" containerID="70e71545f953c2461bad29ce731f98966cccf1b2975d11bd0c87eff5e8ddfe5e" exitCode=0
Sep 09 07:10:52.473364 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:52.472969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-140414780037568" event={"ID":"ecda6810-8fca-4c32-a3df-2fe5cb7d5282","Type":"ContainerDied","Data":"70e71545f953c2461bad29ce731f98966cccf1b2975d11bd0c87eff5e8ddfe5e"}
Sep 09 07:10:53.166032 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.165998    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:10:53.636778 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.636753    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:53.656618 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.656581    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7943-140414780037568_ecda6810-8fca-4c32-a3df-2fe5cb7d5282/test-curl-7943-140414780037568/0.log"
Sep 09 07:10:53.679674 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.679635    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:53.684645 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.684610    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:53.714079 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.714021    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r74pk\" (UniqueName: \"kubernetes.io/projected/ecda6810-8fca-4c32-a3df-2fe5cb7d5282-kube-api-access-r74pk\") pod \"ecda6810-8fca-4c32-a3df-2fe5cb7d5282\" (UID: \"ecda6810-8fca-4c32-a3df-2fe5cb7d5282\") "
Sep 09 07:10:53.716343 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.716309    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecda6810-8fca-4c32-a3df-2fe5cb7d5282-kube-api-access-r74pk" (OuterVolumeSpecName: "kube-api-access-r74pk") pod "ecda6810-8fca-4c32-a3df-2fe5cb7d5282" (UID: "ecda6810-8fca-4c32-a3df-2fe5cb7d5282"). InnerVolumeSpecName "kube-api-access-r74pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:10:53.797431 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.797339    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:53.797720 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.797705    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecda6810-8fca-4c32-a3df-2fe5cb7d5282" containerName="test-curl-7943-140414780037568"
Sep 09 07:10:53.797762 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.797720    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecda6810-8fca-4c32-a3df-2fe5cb7d5282" containerName="test-curl-7943-140414780037568"
Sep 09 07:10:53.797794 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.797778    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecda6810-8fca-4c32-a3df-2fe5cb7d5282" containerName="test-curl-7943-140414780037568"
Sep 09 07:10:53.800849 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.800831    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:53.805129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.805088    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:53.815089 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.815061    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7b9\" (UniqueName: \"kubernetes.io/projected/aad209bd-58f6-4003-ad06-8eadc85df4b8-kube-api-access-qw7b9\") pod \"test-curl-7943-140414780037568\" (UID: \"aad209bd-58f6-4003-ad06-8eadc85df4b8\") " pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:53.815214 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.815186    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74pk\" (UniqueName: \"kubernetes.io/projected/ecda6810-8fca-4c32-a3df-2fe5cb7d5282-kube-api-access-r74pk\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:10:53.915678 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.915645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7b9\" (UniqueName: \"kubernetes.io/projected/aad209bd-58f6-4003-ad06-8eadc85df4b8-kube-api-access-qw7b9\") pod \"test-curl-7943-140414780037568\" (UID: \"aad209bd-58f6-4003-ad06-8eadc85df4b8\") " pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:53.925842 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:53.925808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7b9\" (UniqueName: \"kubernetes.io/projected/aad209bd-58f6-4003-ad06-8eadc85df4b8-kube-api-access-qw7b9\") pod \"test-curl-7943-140414780037568\" (UID: \"aad209bd-58f6-4003-ad06-8eadc85df4b8\") " pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:54.111457 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:54.111364    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:54.381649 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:54.381620    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:54.383440 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:10:54.383407    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad209bd_58f6_4003_ad06_8eadc85df4b8.slice/crio-7dce399d653b38929e3bd48c91bf801b9900822bc98a61a051a7d41fc7f54cdb WatchSource:0}: Error finding container 7dce399d653b38929e3bd48c91bf801b9900822bc98a61a051a7d41fc7f54cdb: Status 404 returned error can't find the container with id 7dce399d653b38929e3bd48c91bf801b9900822bc98a61a051a7d41fc7f54cdb
Sep 09 07:10:54.493021 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:54.492988    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69c9d6b3495c561226f200ed5ab8f2d5c583af6671b87d7f7d5468c0d727d6fd"
Sep 09 07:10:54.493021 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:54.492998    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:54.494464 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:54.494433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-140414780037568" event={"ID":"aad209bd-58f6-4003-ad06-8eadc85df4b8","Type":"ContainerStarted","Data":"7dce399d653b38929e3bd48c91bf801b9900822bc98a61a051a7d41fc7f54cdb"}
Sep 09 07:10:54.504628 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:54.504603    2562 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7943-140414780037568" oldPodUID="ecda6810-8fca-4c32-a3df-2fe5cb7d5282" podUID="aad209bd-58f6-4003-ad06-8eadc85df4b8"
Sep 09 07:10:55.230408 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:55.230379    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecda6810-8fca-4c32-a3df-2fe5cb7d5282" path="/var/lib/kubelet/pods/ecda6810-8fca-4c32-a3df-2fe5cb7d5282/volumes"
Sep 09 07:10:55.502227 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:55.502192    2562 generic.go:356] "Generic (PLEG): container finished" podID="aad209bd-58f6-4003-ad06-8eadc85df4b8" containerID="c9b121317bc05932fc930360a6c9ddf2f25abd24e3b643d4b0d641f14a3b3488" exitCode=0
Sep 09 07:10:55.502426 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:55.502280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7943-140414780037568" event={"ID":"aad209bd-58f6-4003-ad06-8eadc85df4b8","Type":"ContainerDied","Data":"c9b121317bc05932fc930360a6c9ddf2f25abd24e3b643d4b0d641f14a3b3488"}
Sep 09 07:10:56.640841 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:56.640813    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:56.660465 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:56.660441    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7943-140414780037568_aad209bd-58f6-4003-ad06-8eadc85df4b8/test-curl-7943-140414780037568/0.log"
Sep 09 07:10:56.683599 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:56.683572    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:56.685763 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:56.685736    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7943-140414780037568"]
Sep 09 07:10:56.741129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:56.741062    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw7b9\" (UniqueName: \"kubernetes.io/projected/aad209bd-58f6-4003-ad06-8eadc85df4b8-kube-api-access-qw7b9\") pod \"aad209bd-58f6-4003-ad06-8eadc85df4b8\" (UID: \"aad209bd-58f6-4003-ad06-8eadc85df4b8\") "
Sep 09 07:10:56.743113 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:56.743077    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad209bd-58f6-4003-ad06-8eadc85df4b8-kube-api-access-qw7b9" (OuterVolumeSpecName: "kube-api-access-qw7b9") pod "aad209bd-58f6-4003-ad06-8eadc85df4b8" (UID: "aad209bd-58f6-4003-ad06-8eadc85df4b8"). InnerVolumeSpecName "kube-api-access-qw7b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:10:56.841876 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:56.841793    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw7b9\" (UniqueName: \"kubernetes.io/projected/aad209bd-58f6-4003-ad06-8eadc85df4b8-kube-api-access-qw7b9\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:10:57.230619 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:57.230582    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad209bd-58f6-4003-ad06-8eadc85df4b8" path="/var/lib/kubelet/pods/aad209bd-58f6-4003-ad06-8eadc85df4b8/volumes"
Sep 09 07:10:57.516571 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:57.516488    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7943-140414780037568"
Sep 09 07:10:57.516755 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:10:57.516547    2562 scope.go:117] "RemoveContainer" containerID="c9b121317bc05932fc930360a6c9ddf2f25abd24e3b643d4b0d641f14a3b3488"
Sep 09 07:11:02.963452 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.963412    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"]
Sep 09 07:11:02.963933 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.963835    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad209bd-58f6-4003-ad06-8eadc85df4b8" containerName="test-curl-7943-140414780037568"
Sep 09 07:11:02.963933 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.963853    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad209bd-58f6-4003-ad06-8eadc85df4b8" containerName="test-curl-7943-140414780037568"
Sep 09 07:11:02.964069 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.963959    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad209bd-58f6-4003-ad06-8eadc85df4b8" containerName="test-curl-7943-140414780037568"
Sep 09 07:11:02.968643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.968627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:02.971215 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.971192    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0cc2c009\""
Sep 09 07:11:02.974493 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.974471    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"]
Sep 09 07:11:02.977929 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.977912    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:02.981963 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.981937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"]
Sep 09 07:11:02.985125 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:02.985078    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0cc2c009-dockercfg-qk5xx\""
Sep 09 07:11:03.002152 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.002121    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"]
Sep 09 07:11:03.002794 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.002763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljkg\" (UniqueName: \"kubernetes.io/projected/d943df76-65b6-43c2-a62b-cdf2feafa02d-kube-api-access-mljkg\") pod \"payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s\" (UID: \"d943df76-65b6-43c2-a62b-cdf2feafa02d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:03.002916 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.002812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943df76-65b6-43c2-a62b-cdf2feafa02d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s\" (UID: \"d943df76-65b6-43c2-a62b-cdf2feafa02d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:03.002916 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.002853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glv44\" (UniqueName: \"kubernetes.io/projected/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-kube-api-access-glv44\") pod \"payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f\" (UID: \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\") " pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:03.002916 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.002900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-plugins-config-volume\") pod \"payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f\" (UID: \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\") " pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:03.104675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.104623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljkg\" (UniqueName: \"kubernetes.io/projected/d943df76-65b6-43c2-a62b-cdf2feafa02d-kube-api-access-mljkg\") pod \"payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s\" (UID: \"d943df76-65b6-43c2-a62b-cdf2feafa02d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:03.104898 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.104678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943df76-65b6-43c2-a62b-cdf2feafa02d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s\" (UID: \"d943df76-65b6-43c2-a62b-cdf2feafa02d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:03.104898 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.104722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glv44\" (UniqueName: \"kubernetes.io/projected/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-kube-api-access-glv44\") pod \"payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f\" (UID: \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\") " pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:03.104898 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.104769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-plugins-config-volume\") pod \"payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f\" (UID: \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\") " pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:03.106013 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.105984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943df76-65b6-43c2-a62b-cdf2feafa02d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s\" (UID: \"d943df76-65b6-43c2-a62b-cdf2feafa02d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:03.106743 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.106348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-plugins-config-volume\") pod \"payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f\" (UID: \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\") " pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:03.120674 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.120640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glv44\" (UniqueName: \"kubernetes.io/projected/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-kube-api-access-glv44\") pod \"payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f\" (UID: \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\") " pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:03.122321 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.122297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljkg\" (UniqueName: \"kubernetes.io/projected/d943df76-65b6-43c2-a62b-cdf2feafa02d-kube-api-access-mljkg\") pod \"payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s\" (UID: \"d943df76-65b6-43c2-a62b-cdf2feafa02d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:03.279137 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.279024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:03.289673 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.289645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:03.578157 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.578124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"]
Sep 09 07:11:03.578867 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:11:03.578826    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd943df76_65b6_43c2_a62b_cdf2feafa02d.slice/crio-1da421200e80a62ab7c9a36a8a627776f876a078ed031864a3c273409d63ccca WatchSource:0}: Error finding container 1da421200e80a62ab7c9a36a8a627776f876a078ed031864a3c273409d63ccca: Status 404 returned error can't find the container with id 1da421200e80a62ab7c9a36a8a627776f876a078ed031864a3c273409d63ccca
Sep 09 07:11:03.579422 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:11:03.579395    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29303d6a_1ae6_4c34_bcb1_f2e1c29b2069.slice/crio-96616d5c9ccdc6668730b29d0cedc7e59f3fe52c2448ef674ef5677a3d4e3c23 WatchSource:0}: Error finding container 96616d5c9ccdc6668730b29d0cedc7e59f3fe52c2448ef674ef5677a3d4e3c23: Status 404 returned error can't find the container with id 96616d5c9ccdc6668730b29d0cedc7e59f3fe52c2448ef674ef5677a3d4e3c23
Sep 09 07:11:03.580702 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:03.580684    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"]
Sep 09 07:11:04.562386 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:04.562343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f" event={"ID":"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069","Type":"ContainerStarted","Data":"ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac"}
Sep 09 07:11:04.562852 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:04.562433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f" event={"ID":"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069","Type":"ContainerStarted","Data":"96616d5c9ccdc6668730b29d0cedc7e59f3fe52c2448ef674ef5677a3d4e3c23"}
Sep 09 07:11:04.562852 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:04.562466    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:04.564163 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:04.564132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s" event={"ID":"d943df76-65b6-43c2-a62b-cdf2feafa02d","Type":"ContainerStarted","Data":"1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae"}
Sep 09 07:11:04.564273 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:04.564179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s" event={"ID":"d943df76-65b6-43c2-a62b-cdf2feafa02d","Type":"ContainerStarted","Data":"1da421200e80a62ab7c9a36a8a627776f876a078ed031864a3c273409d63ccca"}
Sep 09 07:11:04.564273 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:04.564216    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:04.582891 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:04.582826    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f" podStartSLOduration=2.582810565 podStartE2EDuration="2.582810565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:11:04.581735425 +0000 UTC m=+1187.801976538" watchObservedRunningTime="2026-09-09 07:11:04.582810565 +0000 UTC m=+1187.803051678"
Sep 09 07:11:04.601984 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:04.601931    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s" podStartSLOduration=2.601915354 podStartE2EDuration="2.601915354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:11:04.599382956 +0000 UTC m=+1187.819624074" watchObservedRunningTime="2026-09-09 07:11:04.601915354 +0000 UTC m=+1187.822156496"
Sep 09 07:11:11.870894 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:11.870855    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:11:15.571982 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:15.571951    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:11:15.572397 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:15.572241    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:11:17.234374 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:17.234345    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:11:17.241374 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:17.241352    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:11:17.322586 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:17.322556    2562 scope.go:117] "RemoveContainer" containerID="b51b82fd900609766a3bb667bc8df601cdf5d90c680172ab6bbd88bbf6516078"
Sep 09 07:11:29.065289 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:29.065255    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:11:32.617570 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.617535    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"]
Sep 09 07:11:32.623256 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.623230    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.625568 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.625550    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0cc2c009\"/\"openshift-service-ca.crt\""
Sep 09 07:11:32.626454 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.626433    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0cc2c009\"/\"default-dockercfg-fxmsx\""
Sep 09 07:11:32.626570 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.626476    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0cc2c009\"/\"kube-root-ca.crt\""
Sep 09 07:11:32.626691 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.626482    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0cc2c009\"/\"e2e-lifecycle-model-0cc2c009-kserve-self-signed-certs\""
Sep 09 07:11:32.630049 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.630027    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"]
Sep 09 07:11:32.787753 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.787715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tls-certs\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.787944 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.787778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndx67\" (UniqueName: \"kubernetes.io/projected/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-kube-api-access-ndx67\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.787944 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.787877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-home\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.787944 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.787911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-model-cache\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.787944 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.787939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-dshm\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.788167 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.787984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tmp-dir\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889344 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-home\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889344 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-model-cache\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889344 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889318    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-dshm\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889344 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tmp-dir\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889695 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tls-certs\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889695 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndx67\" (UniqueName: \"kubernetes.io/projected/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-kube-api-access-ndx67\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889805 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-home\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889805 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-model-cache\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.889882 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.889834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tmp-dir\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.891666 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.891641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-dshm\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.891921 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.891903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tls-certs\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.897164 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.897145    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndx67\" (UniqueName: \"kubernetes.io/projected/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-kube-api-access-ndx67\") pod \"e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") " pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:32.934063 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:32.934037    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:33.207337 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:33.207307    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"]
Sep 09 07:11:33.209115 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:11:33.209074    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fbc2fdd_9ba4_4f3a_88e3_bacb532316bd.slice/crio-7efc7d8238b3b60ec20db03f79a624fd56759fa26d767ab276c90b938625c4ed WatchSource:0}: Error finding container 7efc7d8238b3b60ec20db03f79a624fd56759fa26d767ab276c90b938625c4ed: Status 404 returned error can't find the container with id 7efc7d8238b3b60ec20db03f79a624fd56759fa26d767ab276c90b938625c4ed
Sep 09 07:11:33.735192 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:33.735155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx" event={"ID":"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd","Type":"ContainerStarted","Data":"a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29"}
Sep 09 07:11:33.735676 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:33.735201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx" event={"ID":"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd","Type":"ContainerStarted","Data":"7efc7d8238b3b60ec20db03f79a624fd56759fa26d767ab276c90b938625c4ed"}
Sep 09 07:11:33.753085 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:33.753022    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx" podStartSLOduration=1.753006616 podStartE2EDuration="1.753006616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:11:33.75175553 +0000 UTC m=+1216.971996646" watchObservedRunningTime="2026-09-09 07:11:33.753006616 +0000 UTC m=+1216.973247729"
Sep 09 07:11:42.934637 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:42.934599    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:42.935042 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:42.934884    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:42.947217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:42.947193    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:43.806562 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:43.806535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:11:47.763766 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:11:47.763730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:12:06.770227 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:06.770194    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:12:16.436022 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.435941    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"]
Sep 09 07:12:16.436426 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.436238    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f" podUID="29303d6a-1ae6-4c34-bcb1-f2e1c29b2069" containerName="payload-processing" containerID="cri-o://ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac" gracePeriod=30
Sep 09 07:12:16.445643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.445613    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"]
Sep 09 07:12:16.445884 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.445840    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s" podUID="d943df76-65b6-43c2-a62b-cdf2feafa02d" containerName="payload-pre-processing" containerID="cri-o://1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae" gracePeriod=30
Sep 09 07:12:16.700411 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.700354    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:12:16.711212 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.711182    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-plugins-config-volume\") pod \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\" (UID: \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\") "
Sep 09 07:12:16.711343 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.711304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glv44\" (UniqueName: \"kubernetes.io/projected/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-kube-api-access-glv44\") pod \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\" (UID: \"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069\") "
Sep 09 07:12:16.711542 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.711518    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29303d6a-1ae6-4c34-bcb1-f2e1c29b2069" (UID: "29303d6a-1ae6-4c34-bcb1-f2e1c29b2069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:12:16.711686 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.711671    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:16.713303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.713281    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-kube-api-access-glv44" (OuterVolumeSpecName: "kube-api-access-glv44") pod "29303d6a-1ae6-4c34-bcb1-f2e1c29b2069" (UID: "29303d6a-1ae6-4c34-bcb1-f2e1c29b2069"). InnerVolumeSpecName "kube-api-access-glv44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:12:16.726040 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.725974    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:12:16.812871 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.812826    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943df76-65b6-43c2-a62b-cdf2feafa02d-plugins-config-volume\") pod \"d943df76-65b6-43c2-a62b-cdf2feafa02d\" (UID: \"d943df76-65b6-43c2-a62b-cdf2feafa02d\") "
Sep 09 07:12:16.812871 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.812882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mljkg\" (UniqueName: \"kubernetes.io/projected/d943df76-65b6-43c2-a62b-cdf2feafa02d-kube-api-access-mljkg\") pod \"d943df76-65b6-43c2-a62b-cdf2feafa02d\" (UID: \"d943df76-65b6-43c2-a62b-cdf2feafa02d\") "
Sep 09 07:12:16.813263 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.813242    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glv44\" (UniqueName: \"kubernetes.io/projected/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069-kube-api-access-glv44\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:16.813318 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.813283    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d943df76-65b6-43c2-a62b-cdf2feafa02d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d943df76-65b6-43c2-a62b-cdf2feafa02d" (UID: "d943df76-65b6-43c2-a62b-cdf2feafa02d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:12:16.815014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.814993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d943df76-65b6-43c2-a62b-cdf2feafa02d-kube-api-access-mljkg" (OuterVolumeSpecName: "kube-api-access-mljkg") pod "d943df76-65b6-43c2-a62b-cdf2feafa02d" (UID: "d943df76-65b6-43c2-a62b-cdf2feafa02d"). InnerVolumeSpecName "kube-api-access-mljkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:12:16.913862 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.913816    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943df76-65b6-43c2-a62b-cdf2feafa02d-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:16.913862 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.913857    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mljkg\" (UniqueName: \"kubernetes.io/projected/d943df76-65b6-43c2-a62b-cdf2feafa02d-kube-api-access-mljkg\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:16.996487 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.996405    2562 generic.go:356] "Generic (PLEG): container finished" podID="29303d6a-1ae6-4c34-bcb1-f2e1c29b2069" containerID="ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac" exitCode=0
Sep 09 07:12:16.996487 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.996441    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f" event={"ID":"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069","Type":"ContainerDied","Data":"ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac"}
Sep 09 07:12:16.996672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.996505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f" event={"ID":"29303d6a-1ae6-4c34-bcb1-f2e1c29b2069","Type":"ContainerDied","Data":"96616d5c9ccdc6668730b29d0cedc7e59f3fe52c2448ef674ef5677a3d4e3c23"}
Sep 09 07:12:16.996672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.996454    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"
Sep 09 07:12:16.996672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.996539    2562 scope.go:117] "RemoveContainer" containerID="ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac"
Sep 09 07:12:16.998035 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.998011    2562 generic.go:356] "Generic (PLEG): container finished" podID="d943df76-65b6-43c2-a62b-cdf2feafa02d" containerID="1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae" exitCode=0
Sep 09 07:12:16.998184 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.998055    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"
Sep 09 07:12:16.998184 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.998083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s" event={"ID":"d943df76-65b6-43c2-a62b-cdf2feafa02d","Type":"ContainerDied","Data":"1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae"}
Sep 09 07:12:16.998184 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:16.998136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s" event={"ID":"d943df76-65b6-43c2-a62b-cdf2feafa02d","Type":"ContainerDied","Data":"1da421200e80a62ab7c9a36a8a627776f876a078ed031864a3c273409d63ccca"}
Sep 09 07:12:17.005833 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.005813    2562 scope.go:117] "RemoveContainer" containerID="ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac"
Sep 09 07:12:17.006174 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:12:17.006145    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac\": container with ID starting with ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac not found: ID does not exist" containerID="ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac"
Sep 09 07:12:17.006258 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.006185    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac"} err="failed to get container status \"ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac\": rpc error: code = NotFound desc = could not find container \"ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac\": container with ID starting with ba2c2f87cf2c4cd9a73c74f0bc9eec438dfed1e01e8989ab8536c32f3ef4a9ac not found: ID does not exist"
Sep 09 07:12:17.006258 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.006207    2562 scope.go:117] "RemoveContainer" containerID="1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae"
Sep 09 07:12:17.015294 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.015276    2562 scope.go:117] "RemoveContainer" containerID="1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae"
Sep 09 07:12:17.015533 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:12:17.015516    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae\": container with ID starting with 1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae not found: ID does not exist" containerID="1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae"
Sep 09 07:12:17.015575 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.015538    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae"} err="failed to get container status \"1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae\": rpc error: code = NotFound desc = could not find container \"1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae\": container with ID starting with 1aa10f5e3d0c651358833a1ea6ab81fd53fa255083cc033097ea1b03ec1f70ae not found: ID does not exist"
Sep 09 07:12:17.029994 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.029966    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"]
Sep 09 07:12:17.032707 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.032689    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cc2c009-786cf54569-pgk2f"]
Sep 09 07:12:17.045745 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.045724    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"]
Sep 09 07:12:17.051269 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.051247    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cc2c009-757fc6d446-kmh5s"]
Sep 09 07:12:17.232930 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.232896    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29303d6a-1ae6-4c34-bcb1-f2e1c29b2069" path="/var/lib/kubelet/pods/29303d6a-1ae6-4c34-bcb1-f2e1c29b2069/volumes"
Sep 09 07:12:17.233302 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.233287    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d943df76-65b6-43c2-a62b-cdf2feafa02d" path="/var/lib/kubelet/pods/d943df76-65b6-43c2-a62b-cdf2feafa02d/volumes"
Sep 09 07:12:17.353836 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.353756    2562 scope.go:117] "RemoveContainer" containerID="5399232b9e4ab0e3a3dc86f5e19cdb58c87f1376625f4fdc3d823036171d212b"
Sep 09 07:12:17.362480 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.362455    2562 scope.go:117] "RemoveContainer" containerID="d27e8a329d02f1d9786a3ef90e9d7e9017ccf49aff1d173999d1825e21dc635b"
Sep 09 07:12:17.370399 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:17.370369    2562 scope.go:117] "RemoveContainer" containerID="ca18ba04c6b1a4da49f588ece0e51208b146b0213f6fd29badc1669b75346ca8"
Sep 09 07:12:21.430092 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:21.430058    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"]
Sep 09 07:12:24.580307 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:24.580265    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:12:25.255433 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:25.255398    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"]
Sep 09 07:12:26.910417 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.910384    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"]
Sep 09 07:12:26.910795 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.910715    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29303d6a-1ae6-4c34-bcb1-f2e1c29b2069" containerName="payload-processing"
Sep 09 07:12:26.910795 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.910727    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="29303d6a-1ae6-4c34-bcb1-f2e1c29b2069" containerName="payload-processing"
Sep 09 07:12:26.910795 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.910741    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d943df76-65b6-43c2-a62b-cdf2feafa02d" containerName="payload-pre-processing"
Sep 09 07:12:26.910795 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.910746    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d943df76-65b6-43c2-a62b-cdf2feafa02d" containerName="payload-pre-processing"
Sep 09 07:12:26.910918 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.910809    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d943df76-65b6-43c2-a62b-cdf2feafa02d" containerName="payload-pre-processing"
Sep 09 07:12:26.910918 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.910819    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="29303d6a-1ae6-4c34-bcb1-f2e1c29b2069" containerName="payload-processing"
Sep 09 07:12:26.915414 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.915395    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:26.918790 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.918765    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ea0f1c\"/\"default-dockercfg-lt889\""
Sep 09 07:12:26.918990 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.918807    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ea0f1c\"/\"openshift-service-ca.crt\""
Sep 09 07:12:26.918990 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.918866    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ea0f1c\"/\"kube-root-ca.crt\""
Sep 09 07:12:26.919175 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.918999    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ea0f1c\"/\"sub-test-model-ea0f1c-kserve-self-signed-certs\""
Sep 09 07:12:26.922982 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:26.922800    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"]
Sep 09 07:12:27.005916 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.005880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-model-cache\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.006089 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.005946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbpj\" (UniqueName: \"kubernetes.io/projected/61ec68d4-aff0-4d82-8454-ce643fcd3040-kube-api-access-rfbpj\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.006089 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.005981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-dshm\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.006089 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.006029    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ec68d4-aff0-4d82-8454-ce643fcd3040-tls-certs\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.006089 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.006059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-tmp-dir\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.006279 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.006132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-home\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.107566 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.107532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ec68d4-aff0-4d82-8454-ce643fcd3040-tls-certs\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.107566 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.107572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-tmp-dir\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.107839 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.107602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-home\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.107839 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.107665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-model-cache\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.107839 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.107688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbpj\" (UniqueName: \"kubernetes.io/projected/61ec68d4-aff0-4d82-8454-ce643fcd3040-kube-api-access-rfbpj\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.107839 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.107708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-dshm\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.108045 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.107992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-home\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.108045 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.108036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-model-cache\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.108158 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.108036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-tmp-dir\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.109863 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.109834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-dshm\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.110171 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.110154    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ec68d4-aff0-4d82-8454-ce643fcd3040-tls-certs\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.116455 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.116434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbpj\" (UniqueName: \"kubernetes.io/projected/61ec68d4-aff0-4d82-8454-ce643fcd3040-kube-api-access-rfbpj\") pod \"sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.225647 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.225611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:27.511014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:27.510986    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"]
Sep 09 07:12:27.512602 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:12:27.512577    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61ec68d4_aff0_4d82_8454_ce643fcd3040.slice/crio-01f71b1efaef52a2b910b9b54488509273004641fce08bb70460a9eb6bdc7114 WatchSource:0}: Error finding container 01f71b1efaef52a2b910b9b54488509273004641fce08bb70460a9eb6bdc7114: Status 404 returned error can't find the container with id 01f71b1efaef52a2b910b9b54488509273004641fce08bb70460a9eb6bdc7114
Sep 09 07:12:28.064470 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:28.064425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft" event={"ID":"61ec68d4-aff0-4d82-8454-ce643fcd3040","Type":"ContainerStarted","Data":"91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc"}
Sep 09 07:12:28.064853 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:28.064482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft" event={"ID":"61ec68d4-aff0-4d82-8454-ce643fcd3040","Type":"ContainerStarted","Data":"01f71b1efaef52a2b910b9b54488509273004641fce08bb70460a9eb6bdc7114"}
Sep 09 07:12:28.084067 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:28.084018    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft" podStartSLOduration=2.084004693 podStartE2EDuration="2.084004693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:12:28.081673871 +0000 UTC m=+1271.301914985" watchObservedRunningTime="2026-09-09 07:12:28.084004693 +0000 UTC m=+1271.304245806"
Sep 09 07:12:36.449209 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.449167    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf" podUID="65aca61e-a829-4c8f-94ed-b27fd26da039" containerName="main" containerID="cri-o://d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81" gracePeriod=45
Sep 09 07:12:36.697654 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.697629    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:12:36.798642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.798559    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-home\") pod \"65aca61e-a829-4c8f-94ed-b27fd26da039\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") "
Sep 09 07:12:36.798642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.798595    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65aca61e-a829-4c8f-94ed-b27fd26da039-tls-certs\") pod \"65aca61e-a829-4c8f-94ed-b27fd26da039\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") "
Sep 09 07:12:36.798915 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.798679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bdjm\" (UniqueName: \"kubernetes.io/projected/65aca61e-a829-4c8f-94ed-b27fd26da039-kube-api-access-6bdjm\") pod \"65aca61e-a829-4c8f-94ed-b27fd26da039\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") "
Sep 09 07:12:36.798915 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.798710    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-dshm\") pod \"65aca61e-a829-4c8f-94ed-b27fd26da039\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") "
Sep 09 07:12:36.798915 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.798835    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-tmp-dir\") pod \"65aca61e-a829-4c8f-94ed-b27fd26da039\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") "
Sep 09 07:12:36.799098 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.798922    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-model-cache\") pod \"65aca61e-a829-4c8f-94ed-b27fd26da039\" (UID: \"65aca61e-a829-4c8f-94ed-b27fd26da039\") "
Sep 09 07:12:36.799098 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.798837    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-home" (OuterVolumeSpecName: "home") pod "65aca61e-a829-4c8f-94ed-b27fd26da039" (UID: "65aca61e-a829-4c8f-94ed-b27fd26da039"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:12:36.799098 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.799029    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65aca61e-a829-4c8f-94ed-b27fd26da039" (UID: "65aca61e-a829-4c8f-94ed-b27fd26da039"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:12:36.799290 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.799236    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-model-cache" (OuterVolumeSpecName: "model-cache") pod "65aca61e-a829-4c8f-94ed-b27fd26da039" (UID: "65aca61e-a829-4c8f-94ed-b27fd26da039"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:12:36.799352 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.799298    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:36.799352 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.799315    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:36.801010 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.800985    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-dshm" (OuterVolumeSpecName: "dshm") pod "65aca61e-a829-4c8f-94ed-b27fd26da039" (UID: "65aca61e-a829-4c8f-94ed-b27fd26da039"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:12:36.801010 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.801001    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65aca61e-a829-4c8f-94ed-b27fd26da039-kube-api-access-6bdjm" (OuterVolumeSpecName: "kube-api-access-6bdjm") pod "65aca61e-a829-4c8f-94ed-b27fd26da039" (UID: "65aca61e-a829-4c8f-94ed-b27fd26da039"). InnerVolumeSpecName "kube-api-access-6bdjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:12:36.801192 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.801001    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65aca61e-a829-4c8f-94ed-b27fd26da039-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65aca61e-a829-4c8f-94ed-b27fd26da039" (UID: "65aca61e-a829-4c8f-94ed-b27fd26da039"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:12:36.899787 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.899760    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bdjm\" (UniqueName: \"kubernetes.io/projected/65aca61e-a829-4c8f-94ed-b27fd26da039-kube-api-access-6bdjm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:36.899787 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.899783    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:36.899787 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.899792    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65aca61e-a829-4c8f-94ed-b27fd26da039-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:36.899994 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:36.899801    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65aca61e-a829-4c8f-94ed-b27fd26da039-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:37.118832 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.118739    2562 generic.go:356] "Generic (PLEG): container finished" podID="65aca61e-a829-4c8f-94ed-b27fd26da039" containerID="d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81" exitCode=0
Sep 09 07:12:37.118832 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.118780    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"
Sep 09 07:12:37.118832 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.118822    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf" event={"ID":"65aca61e-a829-4c8f-94ed-b27fd26da039","Type":"ContainerDied","Data":"d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81"}
Sep 09 07:12:37.119097 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.118861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf" event={"ID":"65aca61e-a829-4c8f-94ed-b27fd26da039","Type":"ContainerDied","Data":"cb8d14b37c80b77a62614e9feebb193d16d7fbddfaba3efee2c2ddd95a7755cc"}
Sep 09 07:12:37.119097 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.118880    2562 scope.go:117] "RemoveContainer" containerID="d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81"
Sep 09 07:12:37.128434 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.128418    2562 scope.go:117] "RemoveContainer" containerID="d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81"
Sep 09 07:12:37.128691 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:12:37.128675    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81\": container with ID starting with d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81 not found: ID does not exist" containerID="d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81"
Sep 09 07:12:37.128741 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.128699    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81"} err="failed to get container status \"d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81\": rpc error: code = NotFound desc = could not find container \"d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81\": container with ID starting with d75ed447c91d7e7e510fa5fa0746bc6224cc691ec2d595750a012bb77257ba81 not found: ID does not exist"
Sep 09 07:12:37.141843 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.141821    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"]
Sep 09 07:12:37.148040 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.148021    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/auth-test-model-988b24-kserve-64f4864659-dtczf"]
Sep 09 07:12:37.230677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.230644    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65aca61e-a829-4c8f-94ed-b27fd26da039" path="/var/lib/kubelet/pods/65aca61e-a829-4c8f-94ed-b27fd26da039/volumes"
Sep 09 07:12:37.230931 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.230918    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:37.230987 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.230951    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:37.239361 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:37.239338    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:38.137011 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:38.136973    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:12:39.276283 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:39.276242    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:12:40.274809 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.274744    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx" podUID="4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" containerName="main" containerID="cri-o://a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29" gracePeriod=45
Sep 09 07:12:40.526285 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.526226    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:12:40.633611 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.633566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndx67\" (UniqueName: \"kubernetes.io/projected/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-kube-api-access-ndx67\") pod \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") "
Sep 09 07:12:40.633611 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.633621    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-home\") pod \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") "
Sep 09 07:12:40.633870 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.633653    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tls-certs\") pod \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") "
Sep 09 07:12:40.633870 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.633710    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-model-cache\") pod \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") "
Sep 09 07:12:40.633870 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.633725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-dshm\") pod \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") "
Sep 09 07:12:40.633870 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.633757    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tmp-dir\") pod \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\" (UID: \"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd\") "
Sep 09 07:12:40.634079 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.633970    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-home" (OuterVolumeSpecName: "home") pod "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" (UID: "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:12:40.634079 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.633981    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" (UID: "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:12:40.634079 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.634066    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" (UID: "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:12:40.635810 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.635785    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-dshm" (OuterVolumeSpecName: "dshm") pod "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" (UID: "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:12:40.636189 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.636169    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" (UID: "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:12:40.636189 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.636183    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-kube-api-access-ndx67" (OuterVolumeSpecName: "kube-api-access-ndx67") pod "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" (UID: "4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd"). InnerVolumeSpecName "kube-api-access-ndx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:12:40.735134 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.735074    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:40.735134 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.735131    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:40.735134 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.735143    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:40.735134 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.735151    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:40.735406 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.735158    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:40.735406 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:40.735167    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndx67\" (UniqueName: \"kubernetes.io/projected/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd-kube-api-access-ndx67\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:12:41.144202 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.144158    2562 generic.go:356] "Generic (PLEG): container finished" podID="4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" containerID="a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29" exitCode=0
Sep 09 07:12:41.144202 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.144200    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"
Sep 09 07:12:41.144442 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.144246    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx" event={"ID":"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd","Type":"ContainerDied","Data":"a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29"}
Sep 09 07:12:41.144442 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.144287    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx" event={"ID":"4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd","Type":"ContainerDied","Data":"7efc7d8238b3b60ec20db03f79a624fd56759fa26d767ab276c90b938625c4ed"}
Sep 09 07:12:41.144442 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.144304    2562 scope.go:117] "RemoveContainer" containerID="a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29"
Sep 09 07:12:41.153611 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.153592    2562 scope.go:117] "RemoveContainer" containerID="a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29"
Sep 09 07:12:41.153892 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:12:41.153869    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29\": container with ID starting with a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29 not found: ID does not exist" containerID="a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29"
Sep 09 07:12:41.153968 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.153902    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29"} err="failed to get container status \"a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29\": rpc error: code = NotFound desc = could not find container \"a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29\": container with ID starting with a1326b52fdb53b455a12333ac67f14e82af25a745ea537e3dfc126c507d98d29 not found: ID does not exist"
Sep 09 07:12:41.166115 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.166084    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"]
Sep 09 07:12:41.170842 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.170822    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0cc2c009/e2e-lifecycle-model-0cc2c009-kserve-5cbc8ff59f-gjxkx"]
Sep 09 07:12:41.229508 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:41.229477    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" path="/var/lib/kubelet/pods/4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd/volumes"
Sep 09 07:12:53.930000 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.929965    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"]
Sep 09 07:12:53.930528 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.930509    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65aca61e-a829-4c8f-94ed-b27fd26da039" containerName="main"
Sep 09 07:12:53.930528 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.930528    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="65aca61e-a829-4c8f-94ed-b27fd26da039" containerName="main"
Sep 09 07:12:53.930672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.930543    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" containerName="main"
Sep 09 07:12:53.930672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.930553    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" containerName="main"
Sep 09 07:12:53.930672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.930637    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="65aca61e-a829-4c8f-94ed-b27fd26da039" containerName="main"
Sep 09 07:12:53.930672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.930649    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fbc2fdd-9ba4-4f3a-88e3-bacb532316bd" containerName="main"
Sep 09 07:12:53.933675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.933656    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:53.936305 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.936285    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-988b24\"/\"openshift-service-ca.crt\""
Sep 09 07:12:53.937279 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.937251    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-988b24\"/\"default-dockercfg-7bjfk\""
Sep 09 07:12:53.937445 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.937257    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-988b24\"/\"kube-root-ca.crt\""
Sep 09 07:12:53.937445 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.937303    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-988b24\"/\"sub-test-model-988b24-kserve-self-signed-certs\""
Sep 09 07:12:53.945929 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:53.945903    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"]
Sep 09 07:12:54.057933 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.057895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-dshm\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.058144 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.057948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-model-cache\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.058144 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.058027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027d7205-2675-4e7c-8cca-142da2c7af86-tls-certs\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.058144 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.058070    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqppj\" (UniqueName: \"kubernetes.io/projected/027d7205-2675-4e7c-8cca-142da2c7af86-kube-api-access-sqppj\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.058144 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.058092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-home\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.058342 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.058192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-tmp-dir\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.158922 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.158883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-tmp-dir\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.159157 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.158933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-dshm\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.159157 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.158966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-model-cache\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.159157 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.158996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027d7205-2675-4e7c-8cca-142da2c7af86-tls-certs\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.159157 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.159154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqppj\" (UniqueName: \"kubernetes.io/projected/027d7205-2675-4e7c-8cca-142da2c7af86-kube-api-access-sqppj\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.159418 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.159179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-home\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.159418 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.159376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-tmp-dir\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.159492 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.159442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-model-cache\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.159551 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.159531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-home\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.161216 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.161196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-dshm\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.161490 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.161473    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027d7205-2675-4e7c-8cca-142da2c7af86-tls-certs\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.169492 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.169460    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqppj\" (UniqueName: \"kubernetes.io/projected/027d7205-2675-4e7c-8cca-142da2c7af86-kube-api-access-sqppj\") pod \"sub-test-model-988b24-kserve-65894f4c8f-2r272\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") " pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.243519 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.243434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:12:54.517168 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:54.517143    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"]
Sep 09 07:12:54.519273 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:12:54.519248    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027d7205_2675_4e7c_8cca_142da2c7af86.slice/crio-cbe0be99be6131fd9ddc91065418181ce3173efde311c07e10392e6dc8f2518d WatchSource:0}: Error finding container cbe0be99be6131fd9ddc91065418181ce3173efde311c07e10392e6dc8f2518d: Status 404 returned error can't find the container with id cbe0be99be6131fd9ddc91065418181ce3173efde311c07e10392e6dc8f2518d
Sep 09 07:12:55.233604 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:55.233571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272" event={"ID":"027d7205-2675-4e7c-8cca-142da2c7af86","Type":"ContainerStarted","Data":"26d18f37e89e04a01437b4f675f69f94972191c7f2cbd4a67e862be843308e61"}
Sep 09 07:12:55.233604 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:55.233610    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272" event={"ID":"027d7205-2675-4e7c-8cca-142da2c7af86","Type":"ContainerStarted","Data":"cbe0be99be6131fd9ddc91065418181ce3173efde311c07e10392e6dc8f2518d"}
Sep 09 07:12:55.254183 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:55.254136    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272" podStartSLOduration=2.254120609 podStartE2EDuration="2.254120609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:12:55.253389181 +0000 UTC m=+1298.473630295" watchObservedRunningTime="2026-09-09 07:12:55.254120609 +0000 UTC m=+1298.474361719"
Sep 09 07:12:58.386487 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:12:58.386453    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:13:04.243985 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:04.243935    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:13:04.244417 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:04.244263    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:13:04.256614 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:04.256587    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:13:04.297000 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:04.296971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:13:17.404189 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:17.404156    2562 scope.go:117] "RemoveContainer" containerID="287ddec0f8703da6d43c7f4578fbf71accc65926de9c3e65d881ba5e6382dfc6"
Sep 09 07:13:17.689854 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:17.689819    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:13:36.902097 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:36.902063    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:13:39.492650 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:39.492614    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"]
Sep 09 07:13:40.367853 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:40.367767    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"]
Sep 09 07:13:49.979136 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:49.979072    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:13:54.512386 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.512346    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft" podUID="61ec68d4-aff0-4d82-8454-ce643fcd3040" containerName="main" containerID="cri-o://91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc" gracePeriod=45
Sep 09 07:13:54.762129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.762096    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:13:54.826640 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.826571    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ec68d4-aff0-4d82-8454-ce643fcd3040-tls-certs\") pod \"61ec68d4-aff0-4d82-8454-ce643fcd3040\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") "
Sep 09 07:13:54.826640 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.826614    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-dshm\") pod \"61ec68d4-aff0-4d82-8454-ce643fcd3040\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") "
Sep 09 07:13:54.826846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.826652    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-model-cache\") pod \"61ec68d4-aff0-4d82-8454-ce643fcd3040\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") "
Sep 09 07:13:54.826846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.826672    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-tmp-dir\") pod \"61ec68d4-aff0-4d82-8454-ce643fcd3040\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") "
Sep 09 07:13:54.826846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.826729    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfbpj\" (UniqueName: \"kubernetes.io/projected/61ec68d4-aff0-4d82-8454-ce643fcd3040-kube-api-access-rfbpj\") pod \"61ec68d4-aff0-4d82-8454-ce643fcd3040\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") "
Sep 09 07:13:54.826846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.826751    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-home\") pod \"61ec68d4-aff0-4d82-8454-ce643fcd3040\" (UID: \"61ec68d4-aff0-4d82-8454-ce643fcd3040\") "
Sep 09 07:13:54.827051 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.826960    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-model-cache" (OuterVolumeSpecName: "model-cache") pod "61ec68d4-aff0-4d82-8454-ce643fcd3040" (UID: "61ec68d4-aff0-4d82-8454-ce643fcd3040"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:13:54.827051 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.826975    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61ec68d4-aff0-4d82-8454-ce643fcd3040" (UID: "61ec68d4-aff0-4d82-8454-ce643fcd3040"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:13:54.827210 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.827191    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-home" (OuterVolumeSpecName: "home") pod "61ec68d4-aff0-4d82-8454-ce643fcd3040" (UID: "61ec68d4-aff0-4d82-8454-ce643fcd3040"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:13:54.828922 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.828903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ec68d4-aff0-4d82-8454-ce643fcd3040-kube-api-access-rfbpj" (OuterVolumeSpecName: "kube-api-access-rfbpj") pod "61ec68d4-aff0-4d82-8454-ce643fcd3040" (UID: "61ec68d4-aff0-4d82-8454-ce643fcd3040"). InnerVolumeSpecName "kube-api-access-rfbpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:13:54.829189 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.829172    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ec68d4-aff0-4d82-8454-ce643fcd3040-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61ec68d4-aff0-4d82-8454-ce643fcd3040" (UID: "61ec68d4-aff0-4d82-8454-ce643fcd3040"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:13:54.829189 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.829177    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-dshm" (OuterVolumeSpecName: "dshm") pod "61ec68d4-aff0-4d82-8454-ce643fcd3040" (UID: "61ec68d4-aff0-4d82-8454-ce643fcd3040"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:13:54.927810 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.927780    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:54.927810 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.927804    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:54.927810 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.927813    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfbpj\" (UniqueName: \"kubernetes.io/projected/61ec68d4-aff0-4d82-8454-ce643fcd3040-kube-api-access-rfbpj\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:54.928007 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.927824    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:54.928007 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.927832    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ec68d4-aff0-4d82-8454-ce643fcd3040-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:54.928007 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:54.927841    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ec68d4-aff0-4d82-8454-ce643fcd3040-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:55.337337 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.337301    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"]
Sep 09 07:13:55.337680 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.337667    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ec68d4-aff0-4d82-8454-ce643fcd3040" containerName="main"
Sep 09 07:13:55.337736 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.337682    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ec68d4-aff0-4d82-8454-ce643fcd3040" containerName="main"
Sep 09 07:13:55.337736 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.337734    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ec68d4-aff0-4d82-8454-ce643fcd3040" containerName="main"
Sep 09 07:13:55.340922 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.340900    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.344169 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.344144    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ea0f1c\"/\"rate-test-model-ea0f1c-kserve-self-signed-certs\""
Sep 09 07:13:55.353539 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.353514    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"]
Sep 09 07:13:55.387486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.387446    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272" podUID="027d7205-2675-4e7c-8cca-142da2c7af86" containerName="main" containerID="cri-o://26d18f37e89e04a01437b4f675f69f94972191c7f2cbd4a67e862be843308e61" gracePeriod=45
Sep 09 07:13:55.432152 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.432090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxs8\" (UniqueName: \"kubernetes.io/projected/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-kube-api-access-prxs8\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.432308 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.432164    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tmp-dir\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.432308 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.432187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-dshm\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.432308 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.432233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-home\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.432423 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.432301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tls-certs\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.432423 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.432348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-model-cache\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.532875 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.532843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-home\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.533255 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.532899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tls-certs\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.533255 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.532933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-model-cache\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.533255 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.533005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxs8\" (UniqueName: \"kubernetes.io/projected/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-kube-api-access-prxs8\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.533255 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.533030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tmp-dir\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.533255 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.533051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-dshm\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.533510 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.533294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-home\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.533593 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.533572    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-model-cache\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.533653 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.533614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tmp-dir\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.544570 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.544457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tls-certs\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.548364 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.547376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-dshm\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.555459 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.555321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxs8\" (UniqueName: \"kubernetes.io/projected/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-kube-api-access-prxs8\") pod \"rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") " pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.578637 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.578609    2562 generic.go:356] "Generic (PLEG): container finished" podID="027d7205-2675-4e7c-8cca-142da2c7af86" containerID="26d18f37e89e04a01437b4f675f69f94972191c7f2cbd4a67e862be843308e61" exitCode=0
Sep 09 07:13:55.578794 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.578683    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272" event={"ID":"027d7205-2675-4e7c-8cca-142da2c7af86","Type":"ContainerDied","Data":"26d18f37e89e04a01437b4f675f69f94972191c7f2cbd4a67e862be843308e61"}
Sep 09 07:13:55.580093 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.580072    2562 generic.go:356] "Generic (PLEG): container finished" podID="61ec68d4-aff0-4d82-8454-ce643fcd3040" containerID="91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc" exitCode=0
Sep 09 07:13:55.580243 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.580129    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"
Sep 09 07:13:55.580243 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.580154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft" event={"ID":"61ec68d4-aff0-4d82-8454-ce643fcd3040","Type":"ContainerDied","Data":"91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc"}
Sep 09 07:13:55.580243 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.580216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft" event={"ID":"61ec68d4-aff0-4d82-8454-ce643fcd3040","Type":"ContainerDied","Data":"01f71b1efaef52a2b910b9b54488509273004641fce08bb70460a9eb6bdc7114"}
Sep 09 07:13:55.580243 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.580241    2562 scope.go:117] "RemoveContainer" containerID="91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc"
Sep 09 07:13:55.592638 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.592616    2562 scope.go:117] "RemoveContainer" containerID="91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc"
Sep 09 07:13:55.592923 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:13:55.592904    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc\": container with ID starting with 91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc not found: ID does not exist" containerID="91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc"
Sep 09 07:13:55.593009 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.592929    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc"} err="failed to get container status \"91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc\": rpc error: code = NotFound desc = could not find container \"91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc\": container with ID starting with 91afbb087839cfa08c6ff9cf1a511f4aa2bd0518c8924bc0c298193955d54fbc not found: ID does not exist"
Sep 09 07:13:55.601154 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.601127    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"]
Sep 09 07:13:55.606078 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.606056    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/sub-test-model-ea0f1c-kserve-f5dbfcf8c-tz6ft"]
Sep 09 07:13:55.642909 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.642890    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:13:55.650466 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.650443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:13:55.735080 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735051    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027d7205-2675-4e7c-8cca-142da2c7af86-tls-certs\") pod \"027d7205-2675-4e7c-8cca-142da2c7af86\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") "
Sep 09 07:13:55.735241 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735086    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqppj\" (UniqueName: \"kubernetes.io/projected/027d7205-2675-4e7c-8cca-142da2c7af86-kube-api-access-sqppj\") pod \"027d7205-2675-4e7c-8cca-142da2c7af86\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") "
Sep 09 07:13:55.735241 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735144    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-dshm\") pod \"027d7205-2675-4e7c-8cca-142da2c7af86\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") "
Sep 09 07:13:55.735327 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735240    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-home\") pod \"027d7205-2675-4e7c-8cca-142da2c7af86\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") "
Sep 09 07:13:55.735327 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735293    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-model-cache\") pod \"027d7205-2675-4e7c-8cca-142da2c7af86\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") "
Sep 09 07:13:55.735432 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735350    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-tmp-dir\") pod \"027d7205-2675-4e7c-8cca-142da2c7af86\" (UID: \"027d7205-2675-4e7c-8cca-142da2c7af86\") "
Sep 09 07:13:55.735534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735504    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-home" (OuterVolumeSpecName: "home") pod "027d7205-2675-4e7c-8cca-142da2c7af86" (UID: "027d7205-2675-4e7c-8cca-142da2c7af86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:13:55.735622 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735564    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-model-cache" (OuterVolumeSpecName: "model-cache") pod "027d7205-2675-4e7c-8cca-142da2c7af86" (UID: "027d7205-2675-4e7c-8cca-142da2c7af86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:13:55.735708 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735687    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "027d7205-2675-4e7c-8cca-142da2c7af86" (UID: "027d7205-2675-4e7c-8cca-142da2c7af86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:13:55.735778 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735771    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:55.735820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735785    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:55.735820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.735796    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:55.737980 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.737962    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-dshm" (OuterVolumeSpecName: "dshm") pod "027d7205-2675-4e7c-8cca-142da2c7af86" (UID: "027d7205-2675-4e7c-8cca-142da2c7af86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:13:55.738082 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.737962    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027d7205-2675-4e7c-8cca-142da2c7af86-kube-api-access-sqppj" (OuterVolumeSpecName: "kube-api-access-sqppj") pod "027d7205-2675-4e7c-8cca-142da2c7af86" (UID: "027d7205-2675-4e7c-8cca-142da2c7af86"). InnerVolumeSpecName "kube-api-access-sqppj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:13:55.739097 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.739079    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027d7205-2675-4e7c-8cca-142da2c7af86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "027d7205-2675-4e7c-8cca-142da2c7af86" (UID: "027d7205-2675-4e7c-8cca-142da2c7af86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:13:55.836885 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.836798    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027d7205-2675-4e7c-8cca-142da2c7af86-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:55.836885 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.836828    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqppj\" (UniqueName: \"kubernetes.io/projected/027d7205-2675-4e7c-8cca-142da2c7af86-kube-api-access-sqppj\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:55.836885 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.836837    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/027d7205-2675-4e7c-8cca-142da2c7af86-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:13:55.925901 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:55.925874    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"]
Sep 09 07:13:55.927715 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:13:55.927688    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48a6319c_f67c_4f4f_ac55_8396cf4eccb0.slice/crio-2ef706737da2ab40fe3d22ead980551b384c949b95306acbcbd246df64324aaf WatchSource:0}: Error finding container 2ef706737da2ab40fe3d22ead980551b384c949b95306acbcbd246df64324aaf: Status 404 returned error can't find the container with id 2ef706737da2ab40fe3d22ead980551b384c949b95306acbcbd246df64324aaf
Sep 09 07:13:56.590122 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:56.590069    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"
Sep 09 07:13:56.590571 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:56.590075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272" event={"ID":"027d7205-2675-4e7c-8cca-142da2c7af86","Type":"ContainerDied","Data":"cbe0be99be6131fd9ddc91065418181ce3173efde311c07e10392e6dc8f2518d"}
Sep 09 07:13:56.590571 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:56.590239    2562 scope.go:117] "RemoveContainer" containerID="26d18f37e89e04a01437b4f675f69f94972191c7f2cbd4a67e862be843308e61"
Sep 09 07:13:56.594730 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:56.594647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9" event={"ID":"48a6319c-f67c-4f4f-ac55-8396cf4eccb0","Type":"ContainerStarted","Data":"44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab"}
Sep 09 07:13:56.594861 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:56.594746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9" event={"ID":"48a6319c-f67c-4f4f-ac55-8396cf4eccb0","Type":"ContainerStarted","Data":"2ef706737da2ab40fe3d22ead980551b384c949b95306acbcbd246df64324aaf"}
Sep 09 07:13:56.622635 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:56.622592    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9" podStartSLOduration=1.622579396 podStartE2EDuration="1.622579396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:13:56.62047522 +0000 UTC m=+1359.840716333" watchObservedRunningTime="2026-09-09 07:13:56.622579396 +0000 UTC m=+1359.842820505"
Sep 09 07:13:56.643164 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:56.643138    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"]
Sep 09 07:13:56.648827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:56.648793    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-988b24/sub-test-model-988b24-kserve-65894f4c8f-2r272"]
Sep 09 07:13:57.230246 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:57.230216    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027d7205-2675-4e7c-8cca-142da2c7af86" path="/var/lib/kubelet/pods/027d7205-2675-4e7c-8cca-142da2c7af86/volumes"
Sep 09 07:13:57.230623 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:57.230610    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ec68d4-aff0-4d82-8454-ce643fcd3040" path="/var/lib/kubelet/pods/61ec68d4-aff0-4d82-8454-ce643fcd3040/volumes"
Sep 09 07:13:58.973007 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:13:58.972966    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:14:04.898014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:04.897979    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"]
Sep 09 07:14:04.898459 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:04.898340    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027d7205-2675-4e7c-8cca-142da2c7af86" containerName="main"
Sep 09 07:14:04.898459 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:04.898351    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="027d7205-2675-4e7c-8cca-142da2c7af86" containerName="main"
Sep 09 07:14:04.898459 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:04.898417    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="027d7205-2675-4e7c-8cca-142da2c7af86" containerName="main"
Sep 09 07:14:04.902958 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:04.902938    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:04.905722 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:04.905697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-043f83-openshift-default-dockercfg-s5952\""
Sep 09 07:14:04.914473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:04.914451    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"]
Sep 09 07:14:05.020096 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istiod-ca-cert\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.020263 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-data\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.020263 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-envoy\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.020263 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-certs\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.020393 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-credential-socket\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.020393 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-podinfo\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.020393 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvgn\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-kube-api-access-7pvgn\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.020488 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-socket\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.020488 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.020450    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-token\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.121482 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.121441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istiod-ca-cert\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.121685 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.121566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-data\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.121685 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.121670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-envoy\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.121924 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.121901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-certs\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.122090 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.122057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-credential-socket\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.122533 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.122511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-podinfo\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.122661 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.122645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvgn\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-kube-api-access-7pvgn\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.122782 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.122767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-socket\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.122922 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.122904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-token\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.123866 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.121980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-data\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.123866 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.122283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istiod-ca-cert\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.123866 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.122452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-certs\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.123866 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.122523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-credential-socket\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.123866 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.123417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-socket\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.124934 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.124908    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-envoy\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.126209 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.126186    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-podinfo\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.134281 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.134256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvgn\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-kube-api-access-7pvgn\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.134463 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.134445    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-token\") pod \"e2e-api-a-043f83-openshift-default-5784889576-9wrfw\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") " pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.216577 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.216545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:05.496290 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.496257    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"]
Sep 09 07:14:05.497436 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:14:05.497393    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6396a04_d4bf_43dd_87d3_cfb905d5a996.slice/crio-29b1cf9ad887db8ab2588de7846fb9ec95b9ab4be3a9a4a1e591e2f6fc40359b WatchSource:0}: Error finding container 29b1cf9ad887db8ab2588de7846fb9ec95b9ab4be3a9a4a1e591e2f6fc40359b: Status 404 returned error can't find the container with id 29b1cf9ad887db8ab2588de7846fb9ec95b9ab4be3a9a4a1e591e2f6fc40359b
Sep 09 07:14:05.499786 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.499749    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:14:05.499879 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.499825    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:14:05.499879 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.499853    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:14:05.650666 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.650630    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:14:05.650870 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.650684    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:14:05.651015 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.650988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" event={"ID":"e6396a04-d4bf-43dd-87d3-cfb905d5a996","Type":"ContainerStarted","Data":"65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a"}
Sep 09 07:14:05.651141 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.651045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" event={"ID":"e6396a04-d4bf-43dd-87d3-cfb905d5a996","Type":"ContainerStarted","Data":"29b1cf9ad887db8ab2588de7846fb9ec95b9ab4be3a9a4a1e591e2f6fc40359b"}
Sep 09 07:14:05.668950 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.668918    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:14:05.673186 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:05.673130    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" podStartSLOduration=1.67307467 podStartE2EDuration="1.67307467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:14:05.670375538 +0000 UTC m=+1368.890616653" watchObservedRunningTime="2026-09-09 07:14:05.67307467 +0000 UTC m=+1368.893315777"
Sep 09 07:14:06.217301 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:06.217263    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:06.222296 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:06.222273    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:06.657057 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:06.656955    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:06.657764 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:06.657741    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:06.668002 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:06.667980    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:14:09.668364 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.668331    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"]
Sep 09 07:14:09.672153 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.672130    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:09.676217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.676197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-043f83-dockercfg-9nrc5\""
Sep 09 07:14:09.676333 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.676261    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-043f83\""
Sep 09 07:14:09.684770 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.684745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"]
Sep 09 07:14:09.768527 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.768492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlncm\" (UniqueName: \"kubernetes.io/projected/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-kube-api-access-hlncm\") pod \"payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm\" (UID: \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:09.768693 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.768565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm\" (UID: \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:09.869263 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.869229    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlncm\" (UniqueName: \"kubernetes.io/projected/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-kube-api-access-hlncm\") pod \"payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm\" (UID: \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:09.869480 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.869292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm\" (UID: \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:09.869945 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.869924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm\" (UID: \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:09.879047 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.879016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlncm\" (UniqueName: \"kubernetes.io/projected/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-kube-api-access-hlncm\") pod \"payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm\" (UID: \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:09.984556 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:09.984493    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:10.259234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:10.259204    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"]
Sep 09 07:14:10.260147 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:14:10.260116    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a34317e_bf15_4cef_bb0c_4521b0efdc9a.slice/crio-dc97cfcdc6841b778987882845890a3f26ca2934cb11d85195cd3f41f3caebec WatchSource:0}: Error finding container dc97cfcdc6841b778987882845890a3f26ca2934cb11d85195cd3f41f3caebec: Status 404 returned error can't find the container with id dc97cfcdc6841b778987882845890a3f26ca2934cb11d85195cd3f41f3caebec
Sep 09 07:14:10.684859 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:10.684820    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm" event={"ID":"9a34317e-bf15-4cef-bb0c-4521b0efdc9a","Type":"ContainerStarted","Data":"59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd"}
Sep 09 07:14:10.685258 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:10.684930    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:10.685258 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:10.684959    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm" event={"ID":"9a34317e-bf15-4cef-bb0c-4521b0efdc9a","Type":"ContainerStarted","Data":"dc97cfcdc6841b778987882845890a3f26ca2934cb11d85195cd3f41f3caebec"}
Sep 09 07:14:10.703454 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:10.703405    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm" podStartSLOduration=1.703387873 podStartE2EDuration="1.703387873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:14:10.70151208 +0000 UTC m=+1373.921753193" watchObservedRunningTime="2026-09-09 07:14:10.703387873 +0000 UTC m=+1373.923628987"
Sep 09 07:14:16.276672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:16.276632    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:14:17.433745 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:17.433709    2562 scope.go:117] "RemoveContainer" containerID="3c7fe9d288f6f7b2124d03bac3fabcf2d8f4af898d923d7a0ae7b3c40faf928d"
Sep 09 07:14:17.442540 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:17.442430    2562 scope.go:117] "RemoveContainer" containerID="00c13d45cf35bc3c2473923e671459c570a5038fcfe7f4baac432dd4f430c2b4"
Sep 09 07:14:21.692637 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:21.692608    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:14:26.778296 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:26.778265    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerID="65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a" exitCode=0
Sep 09 07:14:26.778708 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:26.778335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" event={"ID":"e6396a04-d4bf-43dd-87d3-cfb905d5a996","Type":"ContainerDied","Data":"65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a"}
Sep 09 07:14:26.778752 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:26.778720    2562 scope.go:117] "RemoveContainer" containerID="65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a"
Sep 09 07:14:26.779899 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:26.779869    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:14:26.780456 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:26.780421    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:14:26.780726 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:26.780702    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:14:27.790819 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:27.790780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" event={"ID":"e6396a04-d4bf-43dd-87d3-cfb905d5a996","Type":"ContainerStarted","Data":"317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c"}
Sep 09 07:14:28.216929 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:28.216893    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:28.217217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:28.217193    2562 patch_prober.go:28] interesting pod/e2e-api-a-043f83-openshift-default-5784889576-9wrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 07:14:28.217283 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:28.217253    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 07:14:29.217304 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:29.217259    2562 patch_prober.go:28] interesting pod/e2e-api-a-043f83-openshift-default-5784889576-9wrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 07:14:29.217809 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:29.217338    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 07:14:29.772917 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:29.772873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:14:30.217903 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:30.217866    2562 patch_prober.go:28] interesting pod/e2e-api-a-043f83-openshift-default-5784889576-9wrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 07:14:30.218340 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:30.217924    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 07:14:31.217590 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:31.217553    2562 patch_prober.go:28] interesting pod/e2e-api-a-043f83-openshift-default-5784889576-9wrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 07:14:31.217778 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:31.217609    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 07:14:32.217196 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:32.217155    2562 patch_prober.go:28] interesting pod/e2e-api-a-043f83-openshift-default-5784889576-9wrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 09 07:14:32.217627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:32.217216    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 09 07:14:33.221086 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:33.221055    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:33.221607 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:33.221372    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:33.222049 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:33.222033    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:14:39.521309 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.521273    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"]
Sep 09 07:14:39.524374 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.524355    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:39.528155 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.528127    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d8398c\""
Sep 09 07:14:39.528276 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.528184    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d8398c-dockercfg-26hpm\""
Sep 09 07:14:39.541363 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.541339    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"]
Sep 09 07:14:39.561901 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.561874    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4\" (UID: \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:39.562017 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.561996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9ltt\" (UniqueName: \"kubernetes.io/projected/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-kube-api-access-g9ltt\") pod \"payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4\" (UID: \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:39.581474 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.581445    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"]
Sep 09 07:14:39.583893 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.583875    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:39.596577 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.596550    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"]
Sep 09 07:14:39.663486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.663442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46gt9\" (UniqueName: \"kubernetes.io/projected/a2f6282e-da06-464c-af01-376206ba6262-kube-api-access-46gt9\") pod \"payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw\" (UID: \"a2f6282e-da06-464c-af01-376206ba6262\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:39.663697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.663540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9ltt\" (UniqueName: \"kubernetes.io/projected/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-kube-api-access-g9ltt\") pod \"payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4\" (UID: \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:39.663697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.663587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f6282e-da06-464c-af01-376206ba6262-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw\" (UID: \"a2f6282e-da06-464c-af01-376206ba6262\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:39.663697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.663679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4\" (UID: \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:39.664418 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.664396    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4\" (UID: \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:39.674777 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.674745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9ltt\" (UniqueName: \"kubernetes.io/projected/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-kube-api-access-g9ltt\") pod \"payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4\" (UID: \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:39.764274 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.764239    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f6282e-da06-464c-af01-376206ba6262-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw\" (UID: \"a2f6282e-da06-464c-af01-376206ba6262\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:39.764411 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.764389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46gt9\" (UniqueName: \"kubernetes.io/projected/a2f6282e-da06-464c-af01-376206ba6262-kube-api-access-46gt9\") pod \"payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw\" (UID: \"a2f6282e-da06-464c-af01-376206ba6262\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:39.764960 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.764938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f6282e-da06-464c-af01-376206ba6262-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw\" (UID: \"a2f6282e-da06-464c-af01-376206ba6262\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:39.772956 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.772899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46gt9\" (UniqueName: \"kubernetes.io/projected/a2f6282e-da06-464c-af01-376206ba6262-kube-api-access-46gt9\") pod \"payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw\" (UID: \"a2f6282e-da06-464c-af01-376206ba6262\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:39.834857 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.834825    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:39.895333 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:39.894652    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:40.110747 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.110719    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"]
Sep 09 07:14:40.111769 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:14:40.111738    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c364500_f8ca_4f97_ad9f_0b3843d00dd8.slice/crio-32b7752416cbc55a5b8e7a43095d25135e82d39dad320feecf30f648059ed08e WatchSource:0}: Error finding container 32b7752416cbc55a5b8e7a43095d25135e82d39dad320feecf30f648059ed08e: Status 404 returned error can't find the container with id 32b7752416cbc55a5b8e7a43095d25135e82d39dad320feecf30f648059ed08e
Sep 09 07:14:40.170773 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:14:40.170746    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f6282e_da06_464c_af01_376206ba6262.slice/crio-05d89bda9a53e9c3dc0289b52beb62d9064c019bdc43781e68ab2889fe9673f6 WatchSource:0}: Error finding container 05d89bda9a53e9c3dc0289b52beb62d9064c019bdc43781e68ab2889fe9673f6: Status 404 returned error can't find the container with id 05d89bda9a53e9c3dc0289b52beb62d9064c019bdc43781e68ab2889fe9673f6
Sep 09 07:14:40.171060 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.171037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"]
Sep 09 07:14:40.865438 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.865389    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw" event={"ID":"a2f6282e-da06-464c-af01-376206ba6262","Type":"ContainerStarted","Data":"ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7"}
Sep 09 07:14:40.865919 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.865461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw" event={"ID":"a2f6282e-da06-464c-af01-376206ba6262","Type":"ContainerStarted","Data":"05d89bda9a53e9c3dc0289b52beb62d9064c019bdc43781e68ab2889fe9673f6"}
Sep 09 07:14:40.865919 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.865503    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:14:40.867217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.867185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4" event={"ID":"0c364500-f8ca-4f97-ad9f-0b3843d00dd8","Type":"ContainerStarted","Data":"a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169"}
Sep 09 07:14:40.867359 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.867243    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:40.867359 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.867267    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4" event={"ID":"0c364500-f8ca-4f97-ad9f-0b3843d00dd8","Type":"ContainerStarted","Data":"32b7752416cbc55a5b8e7a43095d25135e82d39dad320feecf30f648059ed08e"}
Sep 09 07:14:40.885611 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.885566    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw" podStartSLOduration=1.885552406 podStartE2EDuration="1.885552406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:14:40.881852937 +0000 UTC m=+1404.102094050" watchObservedRunningTime="2026-09-09 07:14:40.885552406 +0000 UTC m=+1404.105793510"
Sep 09 07:14:40.899857 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:40.899814    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4" podStartSLOduration=1.899803109 podStartE2EDuration="1.899803109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:14:40.897691234 +0000 UTC m=+1404.117932350" watchObservedRunningTime="2026-09-09 07:14:40.899803109 +0000 UTC m=+1404.120044222"
Sep 09 07:14:46.966483 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:46.966448    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:14:51.875234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:51.875202    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:14:51.875654 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:14:51.875376    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:15:00.176558 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.176518    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"]
Sep 09 07:15:00.180321 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.180293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"
Sep 09 07:15:00.183090 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.183064    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w"]
Sep 09 07:15:00.184023 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.184000    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:15:00.186471 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.186453    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"]
Sep 09 07:15:00.186628 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.186607    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w"
Sep 09 07:15:00.189904 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.189877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"
Sep 09 07:15:00.191651 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.191630    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"
Sep 09 07:15:00.193456 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.193435    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"]
Sep 09 07:15:00.203083 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.203063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w"
Sep 09 07:15:00.217379 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.217300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"]
Sep 09 07:15:00.220826 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.220800    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w"]
Sep 09 07:15:00.230017 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.229140    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"]
Sep 09 07:15:00.239556 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.239180    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"]
Sep 09 07:15:00.240226 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.240151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"
Sep 09 07:15:00.251945 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.251915    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"
Sep 09 07:15:00.262189 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.262166    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"
Sep 09 07:15:00.488228 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.488200    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"]
Sep 09 07:15:00.489090 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:00.489056    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b384d97_f20f_4e40_9ea3_e93fc37b75e9.slice/crio-a62c3096771aeeb9cb6724e2c5c50b0c5448f148af1b7d71c02f3fd2c65d3062 WatchSource:0}: Error finding container a62c3096771aeeb9cb6724e2c5c50b0c5448f148af1b7d71c02f3fd2c65d3062: Status 404 returned error can't find the container with id a62c3096771aeeb9cb6724e2c5c50b0c5448f148af1b7d71c02f3fd2c65d3062
Sep 09 07:15:00.504863 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.504811    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w"]
Sep 09 07:15:00.505699 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:00.505678    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd979096f_82bd_448b_b195_1a98e1caacbe.slice/crio-9e1cd30f49a04ce5e9a8e3ae23c01960d891df99a3b591bb216758b636b0afc1 WatchSource:0}: Error finding container 9e1cd30f49a04ce5e9a8e3ae23c01960d891df99a3b591bb216758b636b0afc1: Status 404 returned error can't find the container with id 9e1cd30f49a04ce5e9a8e3ae23c01960d891df99a3b591bb216758b636b0afc1
Sep 09 07:15:00.564815 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.564778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"]
Sep 09 07:15:00.565542 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:00.565462    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e328bd_ebfc_42ca_ba50_4a2bb303c66f.slice/crio-aedf39c2bf8d1c4afec39c5e9a94a9b8e4e7a211f1d560fee33ba7135909d50b WatchSource:0}: Error finding container aedf39c2bf8d1c4afec39c5e9a94a9b8e4e7a211f1d560fee33ba7135909d50b: Status 404 returned error can't find the container with id aedf39c2bf8d1c4afec39c5e9a94a9b8e4e7a211f1d560fee33ba7135909d50b
Sep 09 07:15:00.568000 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:00.567972    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b81925_92cc_4089_b323_2f45573bb98c.slice/crio-93d89ba24a68099d18e49f7be6d6441b161b76270ab066c630c6ca0552a5d05f WatchSource:0}: Error finding container 93d89ba24a68099d18e49f7be6d6441b161b76270ab066c630c6ca0552a5d05f: Status 404 returned error can't find the container with id 93d89ba24a68099d18e49f7be6d6441b161b76270ab066c630c6ca0552a5d05f
Sep 09 07:15:00.568772 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.568743    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"]
Sep 09 07:15:00.997572 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.997531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9" event={"ID":"b8b81925-92cc-4089-b323-2f45573bb98c","Type":"ContainerStarted","Data":"c0341c6edc1c685f0eafe78511ddc811fa43037d7c6d32d50ddc09f5f6dc5e2d"}
Sep 09 07:15:00.997754 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.997619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9" event={"ID":"b8b81925-92cc-4089-b323-2f45573bb98c","Type":"ContainerStarted","Data":"93d89ba24a68099d18e49f7be6d6441b161b76270ab066c630c6ca0552a5d05f"}
Sep 09 07:15:00.999232 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.999205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt" event={"ID":"a6e328bd-ebfc-42ca-ba50-4a2bb303c66f","Type":"ContainerStarted","Data":"fb07e1786b295ccaf648dbab807e666efb991ed0683c15e4dfa0b0993965f1ba"}
Sep 09 07:15:00.999317 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:00.999253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt" event={"ID":"a6e328bd-ebfc-42ca-ba50-4a2bb303c66f","Type":"ContainerStarted","Data":"aedf39c2bf8d1c4afec39c5e9a94a9b8e4e7a211f1d560fee33ba7135909d50b"}
Sep 09 07:15:01.000731 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.000704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w" event={"ID":"d979096f-82bd-448b-b195-1a98e1caacbe","Type":"ContainerStarted","Data":"2d17862daccc6fcd5b7b3ecfbb0d1bb1bd6bc85f89d7fa7635134299fb5d149a"}
Sep 09 07:15:01.000857 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.000745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w" event={"ID":"d979096f-82bd-448b-b195-1a98e1caacbe","Type":"ContainerStarted","Data":"9e1cd30f49a04ce5e9a8e3ae23c01960d891df99a3b591bb216758b636b0afc1"}
Sep 09 07:15:01.002207 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.002182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm" event={"ID":"8b384d97-f20f-4e40-9ea3-e93fc37b75e9","Type":"ContainerStarted","Data":"f3ccd31876516aeb2f38d6db929f69475e7d1df790d327ff313cd6086f2892d9"}
Sep 09 07:15:01.002301 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.002231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm" event={"ID":"8b384d97-f20f-4e40-9ea3-e93fc37b75e9","Type":"ContainerStarted","Data":"a62c3096771aeeb9cb6724e2c5c50b0c5448f148af1b7d71c02f3fd2c65d3062"}
Sep 09 07:15:01.013292 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.013240    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9" podStartSLOduration=1.01322128 podStartE2EDuration="1.01322128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:01.012503242 +0000 UTC m=+1424.232744354" watchObservedRunningTime="2026-09-09 07:15:01.01322128 +0000 UTC m=+1424.233462391"
Sep 09 07:15:01.026792 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.026702    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w" podStartSLOduration=1.026685599 podStartE2EDuration="1.026685599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:01.026281751 +0000 UTC m=+1424.246522864" watchObservedRunningTime="2026-09-09 07:15:01.026685599 +0000 UTC m=+1424.246926713"
Sep 09 07:15:01.044965 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.044915    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm" podStartSLOduration=1.044902054 podStartE2EDuration="1.044902054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:01.042562772 +0000 UTC m=+1424.262803891" watchObservedRunningTime="2026-09-09 07:15:01.044902054 +0000 UTC m=+1424.265143166"
Sep 09 07:15:01.058003 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.057961    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt" podStartSLOduration=1.057947741 podStartE2EDuration="1.057947741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:01.056442448 +0000 UTC m=+1424.276683561" watchObservedRunningTime="2026-09-09 07:15:01.057947741 +0000 UTC m=+1424.278188853"
Sep 09 07:15:01.164443 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:01.164405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:15:02.012593 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:02.012554    2562 generic.go:356] "Generic (PLEG): container finished" podID="b8b81925-92cc-4089-b323-2f45573bb98c" containerID="c0341c6edc1c685f0eafe78511ddc811fa43037d7c6d32d50ddc09f5f6dc5e2d" exitCode=0
Sep 09 07:15:02.013046 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:02.012642    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9" event={"ID":"b8b81925-92cc-4089-b323-2f45573bb98c","Type":"ContainerDied","Data":"c0341c6edc1c685f0eafe78511ddc811fa43037d7c6d32d50ddc09f5f6dc5e2d"}
Sep 09 07:15:02.014292 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:02.014268    2562 generic.go:356] "Generic (PLEG): container finished" podID="a6e328bd-ebfc-42ca-ba50-4a2bb303c66f" containerID="fb07e1786b295ccaf648dbab807e666efb991ed0683c15e4dfa0b0993965f1ba" exitCode=0
Sep 09 07:15:02.014405 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:02.014343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt" event={"ID":"a6e328bd-ebfc-42ca-ba50-4a2bb303c66f","Type":"ContainerDied","Data":"fb07e1786b295ccaf648dbab807e666efb991ed0683c15e4dfa0b0993965f1ba"}
Sep 09 07:15:02.015858 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:02.015794    2562 generic.go:356] "Generic (PLEG): container finished" podID="d979096f-82bd-448b-b195-1a98e1caacbe" containerID="2d17862daccc6fcd5b7b3ecfbb0d1bb1bd6bc85f89d7fa7635134299fb5d149a" exitCode=0
Sep 09 07:15:02.015956 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:02.015873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w" event={"ID":"d979096f-82bd-448b-b195-1a98e1caacbe","Type":"ContainerDied","Data":"2d17862daccc6fcd5b7b3ecfbb0d1bb1bd6bc85f89d7fa7635134299fb5d149a"}
Sep 09 07:15:02.017448 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:02.017428    2562 generic.go:356] "Generic (PLEG): container finished" podID="8b384d97-f20f-4e40-9ea3-e93fc37b75e9" containerID="f3ccd31876516aeb2f38d6db929f69475e7d1df790d327ff313cd6086f2892d9" exitCode=0
Sep 09 07:15:02.017543 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:02.017467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm" event={"ID":"8b384d97-f20f-4e40-9ea3-e93fc37b75e9","Type":"ContainerDied","Data":"f3ccd31876516aeb2f38d6db929f69475e7d1df790d327ff313cd6086f2892d9"}
Sep 09 07:15:03.174810 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:03.174777    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"
Sep 09 07:15:03.214097 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:03.214068    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"
Sep 09 07:15:03.240031 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:03.240006    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"
Sep 09 07:15:03.246045 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:03.245899    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w"
Sep 09 07:15:04.030707 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.030680    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w"
Sep 09 07:15:04.030935 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.030727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815635-sn79w" event={"ID":"d979096f-82bd-448b-b195-1a98e1caacbe","Type":"ContainerDied","Data":"9e1cd30f49a04ce5e9a8e3ae23c01960d891df99a3b591bb216758b636b0afc1"}
Sep 09 07:15:04.030935 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.030779    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e1cd30f49a04ce5e9a8e3ae23c01960d891df99a3b591bb216758b636b0afc1"
Sep 09 07:15:04.032155 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.032134    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"
Sep 09 07:15:04.032283 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.032167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm" event={"ID":"8b384d97-f20f-4e40-9ea3-e93fc37b75e9","Type":"ContainerDied","Data":"a62c3096771aeeb9cb6724e2c5c50b0c5448f148af1b7d71c02f3fd2c65d3062"}
Sep 09 07:15:04.032283 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.032208    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a62c3096771aeeb9cb6724e2c5c50b0c5448f148af1b7d71c02f3fd2c65d3062"
Sep 09 07:15:04.033681 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.033652    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"
Sep 09 07:15:04.033807 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.033685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9" event={"ID":"b8b81925-92cc-4089-b323-2f45573bb98c","Type":"ContainerDied","Data":"93d89ba24a68099d18e49f7be6d6441b161b76270ab066c630c6ca0552a5d05f"}
Sep 09 07:15:04.033807 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.033726    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93d89ba24a68099d18e49f7be6d6441b161b76270ab066c630c6ca0552a5d05f"
Sep 09 07:15:04.035201 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.035179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt" event={"ID":"a6e328bd-ebfc-42ca-ba50-4a2bb303c66f","Type":"ContainerDied","Data":"aedf39c2bf8d1c4afec39c5e9a94a9b8e4e7a211f1d560fee33ba7135909d50b"}
Sep 09 07:15:04.035297 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.035201    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"
Sep 09 07:15:04.035297 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:04.035220    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aedf39c2bf8d1c4afec39c5e9a94a9b8e4e7a211f1d560fee33ba7135909d50b"
Sep 09 07:15:06.742605 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.742554    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"]
Sep 09 07:15:06.743178 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743157    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b81925-92cc-4089-b323-2f45573bb98c" containerName="cleanup"
Sep 09 07:15:06.743178 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743179    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b81925-92cc-4089-b323-2f45573bb98c" containerName="cleanup"
Sep 09 07:15:06.743336 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743200    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b384d97-f20f-4e40-9ea3-e93fc37b75e9" containerName="cleanup"
Sep 09 07:15:06.743336 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743209    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b384d97-f20f-4e40-9ea3-e93fc37b75e9" containerName="cleanup"
Sep 09 07:15:06.743336 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743221    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d979096f-82bd-448b-b195-1a98e1caacbe" containerName="cleanup"
Sep 09 07:15:06.743336 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743229    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d979096f-82bd-448b-b195-1a98e1caacbe" containerName="cleanup"
Sep 09 07:15:06.743336 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743246    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e328bd-ebfc-42ca-ba50-4a2bb303c66f" containerName="cleanup"
Sep 09 07:15:06.743336 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743254    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e328bd-ebfc-42ca-ba50-4a2bb303c66f" containerName="cleanup"
Sep 09 07:15:06.743620 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743362    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e328bd-ebfc-42ca-ba50-4a2bb303c66f" containerName="cleanup"
Sep 09 07:15:06.743620 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743375    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b81925-92cc-4089-b323-2f45573bb98c" containerName="cleanup"
Sep 09 07:15:06.743620 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743385    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b384d97-f20f-4e40-9ea3-e93fc37b75e9" containerName="cleanup"
Sep 09 07:15:06.743620 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.743400    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d979096f-82bd-448b-b195-1a98e1caacbe" containerName="cleanup"
Sep 09 07:15:06.746706 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.746686    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"
Sep 09 07:15:06.750874 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.750853    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:15:06.750971 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.750934    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:15:06.755735 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.755712    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"]
Sep 09 07:15:06.825232 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.825203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1e3b9-00d7-41e2-a940-b727a1bf0d26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-043f83-2lp9n\" (UID: \"8cd1e3b9-00d7-41e2-a940-b727a1bf0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"
Sep 09 07:15:06.926531 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.926493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1e3b9-00d7-41e2-a940-b727a1bf0d26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-043f83-2lp9n\" (UID: \"8cd1e3b9-00d7-41e2-a940-b727a1bf0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"
Sep 09 07:15:06.927164 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:06.927144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1e3b9-00d7-41e2-a940-b727a1bf0d26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-043f83-2lp9n\" (UID: \"8cd1e3b9-00d7-41e2-a940-b727a1bf0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"
Sep 09 07:15:07.058775 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:07.058675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"
Sep 09 07:15:07.334719 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:07.334682    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"]
Sep 09 07:15:07.335833 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:07.335808    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd1e3b9_00d7_41e2_a940_b727a1bf0d26.slice/crio-674d70455d23252c1d19b3f748e68dc7b92a4db0547fa417be3bd166406d629f WatchSource:0}: Error finding container 674d70455d23252c1d19b3f748e68dc7b92a4db0547fa417be3bd166406d629f: Status 404 returned error can't find the container with id 674d70455d23252c1d19b3f748e68dc7b92a4db0547fa417be3bd166406d629f
Sep 09 07:15:08.059252 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:08.059215    2562 generic.go:356] "Generic (PLEG): container finished" podID="8cd1e3b9-00d7-41e2-a940-b727a1bf0d26" containerID="b5bbe589fe71a63d739e3ca701f4b229992c74f923ba6a1622bcf5b5536b1865" exitCode=0
Sep 09 07:15:08.059712 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:08.059285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n" event={"ID":"8cd1e3b9-00d7-41e2-a940-b727a1bf0d26","Type":"ContainerDied","Data":"b5bbe589fe71a63d739e3ca701f4b229992c74f923ba6a1622bcf5b5536b1865"}
Sep 09 07:15:08.059712 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:08.059359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n" event={"ID":"8cd1e3b9-00d7-41e2-a940-b727a1bf0d26","Type":"ContainerStarted","Data":"674d70455d23252c1d19b3f748e68dc7b92a4db0547fa417be3bd166406d629f"}
Sep 09 07:15:08.762131 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:08.762072    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"]
Sep 09 07:15:09.201958 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:09.201931    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"
Sep 09 07:15:09.348026 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:09.347944    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1e3b9-00d7-41e2-a940-b727a1bf0d26-maas-api-service-ca\") pod \"8cd1e3b9-00d7-41e2-a940-b727a1bf0d26\" (UID: \"8cd1e3b9-00d7-41e2-a940-b727a1bf0d26\") "
Sep 09 07:15:09.348331 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:09.348306    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd1e3b9-00d7-41e2-a940-b727a1bf0d26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cd1e3b9-00d7-41e2-a940-b727a1bf0d26" (UID: "8cd1e3b9-00d7-41e2-a940-b727a1bf0d26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:15:09.448905 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:09.448862    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1e3b9-00d7-41e2-a940-b727a1bf0d26-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:10.074343 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:10.074265    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"
Sep 09 07:15:10.074343 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:10.074292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n" event={"ID":"8cd1e3b9-00d7-41e2-a940-b727a1bf0d26","Type":"ContainerDied","Data":"674d70455d23252c1d19b3f748e68dc7b92a4db0547fa417be3bd166406d629f"}
Sep 09 07:15:10.074343 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:10.074343    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="674d70455d23252c1d19b3f748e68dc7b92a4db0547fa417be3bd166406d629f"
Sep 09 07:15:15.492708 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.492669    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"]
Sep 09 07:15:15.493262 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.493242    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd1e3b9-00d7-41e2-a940-b727a1bf0d26" containerName="revoke-keys"
Sep 09 07:15:15.493357 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.493264    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd1e3b9-00d7-41e2-a940-b727a1bf0d26" containerName="revoke-keys"
Sep 09 07:15:15.493414 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.493388    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd1e3b9-00d7-41e2-a940-b727a1bf0d26" containerName="revoke-keys"
Sep 09 07:15:15.496150 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.496128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:15.498627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.498608    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bea943\""
Sep 09 07:15:15.498771 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.498756    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bea943-dockercfg-zx55t\""
Sep 09 07:15:15.503908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.503881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7c285b-7381-4e4f-9965-4cf55e5570e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw\" (UID: \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:15.503995 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.503965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw627\" (UniqueName: \"kubernetes.io/projected/2b7c285b-7381-4e4f-9965-4cf55e5570e1-kube-api-access-dw627\") pod \"payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw\" (UID: \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:15.509482 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.509456    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"]
Sep 09 07:15:15.605197 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.605162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7c285b-7381-4e4f-9965-4cf55e5570e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw\" (UID: \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:15.605400 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.605214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw627\" (UniqueName: \"kubernetes.io/projected/2b7c285b-7381-4e4f-9965-4cf55e5570e1-kube-api-access-dw627\") pod \"payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw\" (UID: \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:15.605918 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.605894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7c285b-7381-4e4f-9965-4cf55e5570e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw\" (UID: \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:15.612901 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.612875    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw627\" (UniqueName: \"kubernetes.io/projected/2b7c285b-7381-4e4f-9965-4cf55e5570e1-kube-api-access-dw627\") pod \"payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw\" (UID: \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:15.684626 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.684592    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:15:15.806902 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:15.806828    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:16.079921 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:16.079894    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"]
Sep 09 07:15:16.081664 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:16.081637    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7c285b_7381_4e4f_9965_4cf55e5570e1.slice/crio-2fa3ca818ec3ddf1dfe7f0faba7b582872fe56cef00969e1cb623747931c4214 WatchSource:0}: Error finding container 2fa3ca818ec3ddf1dfe7f0faba7b582872fe56cef00969e1cb623747931c4214: Status 404 returned error can't find the container with id 2fa3ca818ec3ddf1dfe7f0faba7b582872fe56cef00969e1cb623747931c4214
Sep 09 07:15:16.116600 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:16.116572    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw" event={"ID":"2b7c285b-7381-4e4f-9965-4cf55e5570e1","Type":"ContainerStarted","Data":"2fa3ca818ec3ddf1dfe7f0faba7b582872fe56cef00969e1cb623747931c4214"}
Sep 09 07:15:17.126408 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:17.126367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw" event={"ID":"2b7c285b-7381-4e4f-9965-4cf55e5570e1","Type":"ContainerStarted","Data":"fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be"}
Sep 09 07:15:17.126812 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:17.126536    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:17.148702 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:17.148660    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw" podStartSLOduration=2.148646523 podStartE2EDuration="2.148646523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:17.14766665 +0000 UTC m=+1440.367907765" watchObservedRunningTime="2026-09-09 07:15:17.148646523 +0000 UTC m=+1440.368887636"
Sep 09 07:15:17.477452 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:17.477411    2562 scope.go:117] "RemoveContainer" containerID="6fe736a72bae83b11430ac52c900058b3c9a6b6863f16538e4c71b96b99b5a4a"
Sep 09 07:15:22.734941 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:22.734898    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"]
Sep 09 07:15:22.737538 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:22.737515    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-043f83-29815635-n5vbt"]
Sep 09 07:15:22.766270 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:22.766237    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"]
Sep 09 07:15:22.766533 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:22.766504    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm" podUID="9a34317e-bf15-4cef-bb0c-4521b0efdc9a" containerName="payload-processing" containerID="cri-o://59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd" gracePeriod=30
Sep 09 07:15:22.969701 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:22.969666    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"]
Sep 09 07:15:22.975261 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:22.975234    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-043f83-2lp9n"]
Sep 09 07:15:23.025195 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.025169    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:15:23.082967 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.082937    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlncm\" (UniqueName: \"kubernetes.io/projected/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-kube-api-access-hlncm\") pod \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\" (UID: \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\") "
Sep 09 07:15:23.083158 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.082985    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-plugins-config-volume\") pod \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\" (UID: \"9a34317e-bf15-4cef-bb0c-4521b0efdc9a\") "
Sep 09 07:15:23.083419 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.083393    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a34317e-bf15-4cef-bb0c-4521b0efdc9a" (UID: "9a34317e-bf15-4cef-bb0c-4521b0efdc9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:15:23.088746 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.088689    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-kube-api-access-hlncm" (OuterVolumeSpecName: "kube-api-access-hlncm") pod "9a34317e-bf15-4cef-bb0c-4521b0efdc9a" (UID: "9a34317e-bf15-4cef-bb0c-4521b0efdc9a"). InnerVolumeSpecName "kube-api-access-hlncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:15:23.162864 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.162829    2562 generic.go:356] "Generic (PLEG): container finished" podID="9a34317e-bf15-4cef-bb0c-4521b0efdc9a" containerID="59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd" exitCode=0
Sep 09 07:15:23.163069 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.162884    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"
Sep 09 07:15:23.163069 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.162907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm" event={"ID":"9a34317e-bf15-4cef-bb0c-4521b0efdc9a","Type":"ContainerDied","Data":"59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd"}
Sep 09 07:15:23.163069 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.162961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm" event={"ID":"9a34317e-bf15-4cef-bb0c-4521b0efdc9a","Type":"ContainerDied","Data":"dc97cfcdc6841b778987882845890a3f26ca2934cb11d85195cd3f41f3caebec"}
Sep 09 07:15:23.163069 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.162985    2562 scope.go:117] "RemoveContainer" containerID="59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd"
Sep 09 07:15:23.173648 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.173632    2562 scope.go:117] "RemoveContainer" containerID="59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd"
Sep 09 07:15:23.173901 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:15:23.173873    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd\": container with ID starting with 59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd not found: ID does not exist" containerID="59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd"
Sep 09 07:15:23.173945 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.173902    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd"} err="failed to get container status \"59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd\": rpc error: code = NotFound desc = could not find container \"59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd\": container with ID starting with 59a2c8e7c5aa708df5c5c27a5eca809da51216c0c7a197ab63d7342e8ec75fbd not found: ID does not exist"
Sep 09 07:15:23.184314 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.184286    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlncm\" (UniqueName: \"kubernetes.io/projected/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-kube-api-access-hlncm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:23.184314 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.184315    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a34317e-bf15-4cef-bb0c-4521b0efdc9a-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:23.187482 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.187451    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"]
Sep 09 07:15:23.192428 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.192403    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-043f83-694b47b76c-pq5pm"]
Sep 09 07:15:23.230335 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.230307    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd1e3b9-00d7-41e2-a940-b727a1bf0d26" path="/var/lib/kubelet/pods/8cd1e3b9-00d7-41e2-a940-b727a1bf0d26/volumes"
Sep 09 07:15:23.230663 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.230650    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a34317e-bf15-4cef-bb0c-4521b0efdc9a" path="/var/lib/kubelet/pods/9a34317e-bf15-4cef-bb0c-4521b0efdc9a/volumes"
Sep 09 07:15:23.230895 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.230884    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e328bd-ebfc-42ca-ba50-4a2bb303c66f" path="/var/lib/kubelet/pods/a6e328bd-ebfc-42ca-ba50-4a2bb303c66f/volumes"
Sep 09 07:15:23.782227 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:23.782181    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9" podUID="48a6319c-f67c-4f4f-ac55-8396cf4eccb0" containerName="main" containerID="cri-o://44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab" gracePeriod=45
Sep 09 07:15:24.022671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.022645    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:15:24.093034 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.092958    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tls-certs\") pod \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") "
Sep 09 07:15:24.093034 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093001    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prxs8\" (UniqueName: \"kubernetes.io/projected/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-kube-api-access-prxs8\") pod \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") "
Sep 09 07:15:24.093034 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093030    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-dshm\") pod \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") "
Sep 09 07:15:24.093340 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093066    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-home\") pod \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") "
Sep 09 07:15:24.093340 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093128    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tmp-dir\") pod \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") "
Sep 09 07:15:24.093340 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093245    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-model-cache\") pod \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\" (UID: \"48a6319c-f67c-4f4f-ac55-8396cf4eccb0\") "
Sep 09 07:15:24.093526 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093380    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-home" (OuterVolumeSpecName: "home") pod "48a6319c-f67c-4f4f-ac55-8396cf4eccb0" (UID: "48a6319c-f67c-4f4f-ac55-8396cf4eccb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:24.093526 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093397    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48a6319c-f67c-4f4f-ac55-8396cf4eccb0" (UID: "48a6319c-f67c-4f4f-ac55-8396cf4eccb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:24.093631 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093532    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-model-cache" (OuterVolumeSpecName: "model-cache") pod "48a6319c-f67c-4f4f-ac55-8396cf4eccb0" (UID: "48a6319c-f67c-4f4f-ac55-8396cf4eccb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:24.093692 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093642    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:24.093692 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093655    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:24.093692 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.093663    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:24.095144 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.095124    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-dshm" (OuterVolumeSpecName: "dshm") pod "48a6319c-f67c-4f4f-ac55-8396cf4eccb0" (UID: "48a6319c-f67c-4f4f-ac55-8396cf4eccb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:24.095242 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.095161    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-kube-api-access-prxs8" (OuterVolumeSpecName: "kube-api-access-prxs8") pod "48a6319c-f67c-4f4f-ac55-8396cf4eccb0" (UID: "48a6319c-f67c-4f4f-ac55-8396cf4eccb0"). InnerVolumeSpecName "kube-api-access-prxs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:15:24.095242 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.095232    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48a6319c-f67c-4f4f-ac55-8396cf4eccb0" (UID: "48a6319c-f67c-4f4f-ac55-8396cf4eccb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:15:24.169960 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.169923    2562 generic.go:356] "Generic (PLEG): container finished" podID="48a6319c-f67c-4f4f-ac55-8396cf4eccb0" containerID="44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab" exitCode=0
Sep 09 07:15:24.170180 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.169967    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"
Sep 09 07:15:24.170180 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.170003    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9" event={"ID":"48a6319c-f67c-4f4f-ac55-8396cf4eccb0","Type":"ContainerDied","Data":"44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab"}
Sep 09 07:15:24.170180 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.170054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9" event={"ID":"48a6319c-f67c-4f4f-ac55-8396cf4eccb0","Type":"ContainerDied","Data":"2ef706737da2ab40fe3d22ead980551b384c949b95306acbcbd246df64324aaf"}
Sep 09 07:15:24.170180 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.170073    2562 scope.go:117] "RemoveContainer" containerID="44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab"
Sep 09 07:15:24.180187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.180168    2562 scope.go:117] "RemoveContainer" containerID="44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab"
Sep 09 07:15:24.180463 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:15:24.180443    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab\": container with ID starting with 44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab not found: ID does not exist" containerID="44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab"
Sep 09 07:15:24.180530 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.180478    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab"} err="failed to get container status \"44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab\": rpc error: code = NotFound desc = could not find container \"44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab\": container with ID starting with 44f5179e4259b73043337d9c0ba12836d4b4035103fc4dd67b855a288de561ab not found: ID does not exist"
Sep 09 07:15:24.192450 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.192422    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"]
Sep 09 07:15:24.194274 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.194253    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:24.194274 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.194275    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxs8\" (UniqueName: \"kubernetes.io/projected/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-kube-api-access-prxs8\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:24.194407 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.194286    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a6319c-f67c-4f4f-ac55-8396cf4eccb0-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:24.196578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:24.196555    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ea0f1c/rate-test-model-ea0f1c-kserve-59c7fc8695-qq4r9"]
Sep 09 07:15:25.230595 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:25.230561    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a6319c-f67c-4f4f-ac55-8396cf4eccb0" path="/var/lib/kubelet/pods/48a6319c-f67c-4f4f-ac55-8396cf4eccb0/volumes"
Sep 09 07:15:28.134830 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:28.134800    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:15:29.635194 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:29.635158    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"]
Sep 09 07:15:29.635623 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:29.635408    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy" containerID="cri-o://317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c" gracePeriod=30
Sep 09 07:15:30.119724 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.119690    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"]
Sep 09 07:15:30.120181 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.120160    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a6319c-f67c-4f4f-ac55-8396cf4eccb0" containerName="main"
Sep 09 07:15:30.120181 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.120182    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a6319c-f67c-4f4f-ac55-8396cf4eccb0" containerName="main"
Sep 09 07:15:30.120367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.120212    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a34317e-bf15-4cef-bb0c-4521b0efdc9a" containerName="payload-processing"
Sep 09 07:15:30.120367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.120221    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a34317e-bf15-4cef-bb0c-4521b0efdc9a" containerName="payload-processing"
Sep 09 07:15:30.120367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.120330    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a34317e-bf15-4cef-bb0c-4521b0efdc9a" containerName="payload-processing"
Sep 09 07:15:30.120367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.120344    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a6319c-f67c-4f4f-ac55-8396cf4eccb0" containerName="main"
Sep 09 07:15:30.124835 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.124814    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"
Sep 09 07:15:30.127612 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.127506    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:15:30.127612 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.127519    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:15:30.130198 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.130166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"]
Sep 09 07:15:30.258996 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.258958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01d809c-8928-4177-bae2-006913f85dfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8398c-h47tw\" (UID: \"f01d809c-8928-4177-bae2-006913f85dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"
Sep 09 07:15:30.359848 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.359806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01d809c-8928-4177-bae2-006913f85dfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8398c-h47tw\" (UID: \"f01d809c-8928-4177-bae2-006913f85dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"
Sep 09 07:15:30.360505 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.360483    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01d809c-8928-4177-bae2-006913f85dfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8398c-h47tw\" (UID: \"f01d809c-8928-4177-bae2-006913f85dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"
Sep 09 07:15:30.438605 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.438574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"
Sep 09 07:15:30.708269 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:30.708238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"]
Sep 09 07:15:30.709898 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:30.709864    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01d809c_8928_4177_bae2_006913f85dfb.slice/crio-b22d56c6070315326569e3a3a7274be4c07dc676515b9e45c83c21dfea35c350 WatchSource:0}: Error finding container b22d56c6070315326569e3a3a7274be4c07dc676515b9e45c83c21dfea35c350: Status 404 returned error can't find the container with id b22d56c6070315326569e3a3a7274be4c07dc676515b9e45c83c21dfea35c350
Sep 09 07:15:31.221095 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:31.221060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw" event={"ID":"f01d809c-8928-4177-bae2-006913f85dfb","Type":"ContainerStarted","Data":"0db3f446705fc29e418616054ebacceb9afff68cf2c3e6277899329e5da91117"}
Sep 09 07:15:31.221286 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:31.221155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw" event={"ID":"f01d809c-8928-4177-bae2-006913f85dfb","Type":"ContainerStarted","Data":"b22d56c6070315326569e3a3a7274be4c07dc676515b9e45c83c21dfea35c350"}
Sep 09 07:15:31.235914 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:31.235867    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw" podStartSLOduration=1.235853412 podStartE2EDuration="1.235853412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:31.235171168 +0000 UTC m=+1454.455412281" watchObservedRunningTime="2026-09-09 07:15:31.235853412 +0000 UTC m=+1454.456094524"
Sep 09 07:15:32.227927 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:32.227888    2562 generic.go:356] "Generic (PLEG): container finished" podID="f01d809c-8928-4177-bae2-006913f85dfb" containerID="0db3f446705fc29e418616054ebacceb9afff68cf2c3e6277899329e5da91117" exitCode=0
Sep 09 07:15:32.228340 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:32.227972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw" event={"ID":"f01d809c-8928-4177-bae2-006913f85dfb","Type":"ContainerDied","Data":"0db3f446705fc29e418616054ebacceb9afff68cf2c3e6277899329e5da91117"}
Sep 09 07:15:33.223696 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:33.223658    2562 patch_prober.go:28] interesting pod/e2e-api-a-043f83-openshift-default-5784889576-9wrfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 07:15:33.223889 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:33.223728    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 07:15:33.369686 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:33.369662    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"
Sep 09 07:15:33.489839 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:33.489747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01d809c-8928-4177-bae2-006913f85dfb-maas-api-service-ca\") pod \"f01d809c-8928-4177-bae2-006913f85dfb\" (UID: \"f01d809c-8928-4177-bae2-006913f85dfb\") "
Sep 09 07:15:33.490088 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:33.490064    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01d809c-8928-4177-bae2-006913f85dfb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f01d809c-8928-4177-bae2-006913f85dfb" (UID: "f01d809c-8928-4177-bae2-006913f85dfb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:15:33.590427 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:33.590391    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01d809c-8928-4177-bae2-006913f85dfb-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:34.240017 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:34.239981    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"
Sep 09 07:15:34.240216 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:34.240025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw" event={"ID":"f01d809c-8928-4177-bae2-006913f85dfb","Type":"ContainerDied","Data":"b22d56c6070315326569e3a3a7274be4c07dc676515b9e45c83c21dfea35c350"}
Sep 09 07:15:34.240216 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:34.240083    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b22d56c6070315326569e3a3a7274be4c07dc676515b9e45c83c21dfea35c350"
Sep 09 07:15:34.894600 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:34.894578    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:15:35.005098 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005069    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istiod-ca-cert\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005098 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005112    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-data\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005371 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-token\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005371 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005179    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pvgn\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-kube-api-access-7pvgn\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005371 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-podinfo\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005371 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005234    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-envoy\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005371 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-socket\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005371 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005331    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-credential-socket\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-certs\") pod \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\" (UID: \"e6396a04-d4bf-43dd-87d3-cfb905d5a996\") "
Sep 09 07:15:35.005677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005549    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:35.005677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005588    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:15:35.005818 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005729    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:35.005818 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005761    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:35.005818 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005801    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:35.005958 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005854    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.005958 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005874    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.005958 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005888    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.005958 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.005901    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.007484 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.007451    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:15:35.007711 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.007693    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-kube-api-access-7pvgn" (OuterVolumeSpecName: "kube-api-access-7pvgn") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "kube-api-access-7pvgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:15:35.007895 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.007868    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:15:35.007975 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.007890    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6396a04-d4bf-43dd-87d3-cfb905d5a996" (UID: "e6396a04-d4bf-43dd-87d3-cfb905d5a996"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:15:35.106975 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.106950    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.106975 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.106971    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pvgn\" (UniqueName: \"kubernetes.io/projected/e6396a04-d4bf-43dd-87d3-cfb905d5a996-kube-api-access-7pvgn\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.107162 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.106982    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.107162 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.106991    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.107162 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.106999    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6396a04-d4bf-43dd-87d3-cfb905d5a996-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:35.160660 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.160630    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:15:35.250516 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.250485    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerID="317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c" exitCode=0
Sep 09 07:15:35.250642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.250529    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"
Sep 09 07:15:35.250642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.250566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" event={"ID":"e6396a04-d4bf-43dd-87d3-cfb905d5a996","Type":"ContainerDied","Data":"317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c"}
Sep 09 07:15:35.250642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.250618    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw" event={"ID":"e6396a04-d4bf-43dd-87d3-cfb905d5a996","Type":"ContainerDied","Data":"29b1cf9ad887db8ab2588de7846fb9ec95b9ab4be3a9a4a1e591e2f6fc40359b"}
Sep 09 07:15:35.250745 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.250645    2562 scope.go:117] "RemoveContainer" containerID="317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c"
Sep 09 07:15:35.259908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.259891    2562 scope.go:117] "RemoveContainer" containerID="65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a"
Sep 09 07:15:35.268393 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.268371    2562 scope.go:117] "RemoveContainer" containerID="317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c"
Sep 09 07:15:35.268684 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:15:35.268664    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c\": container with ID starting with 317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c not found: ID does not exist" containerID="317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c"
Sep 09 07:15:35.268784 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.268696    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c"} err="failed to get container status \"317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c\": rpc error: code = NotFound desc = could not find container \"317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c\": container with ID starting with 317ba20fd46450adcb7ab69762cf5cfae65d1f02d620710681fae4c7eb79201c not found: ID does not exist"
Sep 09 07:15:35.268784 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.268717    2562 scope.go:117] "RemoveContainer" containerID="65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a"
Sep 09 07:15:35.268973 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:15:35.268958    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a\": container with ID starting with 65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a not found: ID does not exist" containerID="65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a"
Sep 09 07:15:35.269038 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.268980    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a"} err="failed to get container status \"65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a\": rpc error: code = NotFound desc = could not find container \"65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a\": container with ID starting with 65ff4f3249ae29c3c612fdc552824c46ff3f1c4a2335fc46b4258cf840c8142a not found: ID does not exist"
Sep 09 07:15:35.269211 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.269193    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"]
Sep 09 07:15:35.277020 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:35.277001    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-043f83-openshift-default-5784889576-9wrfw"]
Sep 09 07:15:37.229626 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:37.229595    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" path="/var/lib/kubelet/pods/e6396a04-d4bf-43dd-87d3-cfb905d5a996/volumes"
Sep 09 07:15:40.076705 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.076667    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"]
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077047    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01d809c-8928-4177-bae2-006913f85dfb" containerName="revoke-keys"
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077060    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01d809c-8928-4177-bae2-006913f85dfb" containerName="revoke-keys"
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077071    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy"
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077077    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy"
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077089    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy"
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077097    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy"
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077176    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01d809c-8928-4177-bae2-006913f85dfb" containerName="revoke-keys"
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077185    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy"
Sep 09 07:15:40.077234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.077196    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6396a04-d4bf-43dd-87d3-cfb905d5a996" containerName="istio-proxy"
Sep 09 07:15:40.081949 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.081922    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.084677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.084646    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-083e4a-openshift-default-dockercfg-wvx2s\""
Sep 09 07:15:40.094816 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.094791    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"]
Sep 09 07:15:40.254270 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-credential-socket\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.254270 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-token\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.254486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-podinfo\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.254486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-socket\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.254486 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istiod-ca-cert\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.254583 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254487    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-certs\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.254583 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkj9m\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-kube-api-access-fkj9m\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.254678 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-envoy\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.254678 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.254645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-data\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.355826 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.355714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-credential-socket\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.355826 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.355783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-token\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356116 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.355830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-podinfo\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356116 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.355882    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-socket\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356116 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.355921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istiod-ca-cert\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356116 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.355953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-certs\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356116 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.356035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkj9m\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-kube-api-access-fkj9m\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356116 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.356068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-envoy\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356605 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.356156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-data\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356605 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.356170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-credential-socket\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.356605 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.356484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-data\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.357158 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.357132    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istiod-ca-cert\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.357484 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.357455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-certs\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.357920 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.357813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-socket\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.360830 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.360780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-podinfo\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.361263 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.361241    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-envoy\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.367540 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.367499    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-token\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.368003 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.367975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkj9m\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-kube-api-access-fkj9m\") pod \"e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") " pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.393932 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.393903    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:40.677060 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.677035    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"]
Sep 09 07:15:40.678646 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:40.678616    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e58b0b8_ff48_46d1_b7e5_fa7b63295609.slice/crio-b347a68e5cd834696de361996b9894254d759a35fbbdd908306e6a025a3fe897 WatchSource:0}: Error finding container b347a68e5cd834696de361996b9894254d759a35fbbdd908306e6a025a3fe897: Status 404 returned error can't find the container with id b347a68e5cd834696de361996b9894254d759a35fbbdd908306e6a025a3fe897
Sep 09 07:15:40.680763 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.680732    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:15:40.680835 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.680810    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:15:40.680878 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:40.680838    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:15:41.288353 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:41.288317    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" event={"ID":"7e58b0b8-ff48-46d1-b7e5-fa7b63295609","Type":"ContainerStarted","Data":"dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a"}
Sep 09 07:15:41.288732 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:41.288377    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" event={"ID":"7e58b0b8-ff48-46d1-b7e5-fa7b63295609","Type":"ContainerStarted","Data":"b347a68e5cd834696de361996b9894254d759a35fbbdd908306e6a025a3fe897"}
Sep 09 07:15:41.311918 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:41.311868    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" podStartSLOduration=1.311852909 podStartE2EDuration="1.311852909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:41.3089364 +0000 UTC m=+1464.529177512" watchObservedRunningTime="2026-09-09 07:15:41.311852909 +0000 UTC m=+1464.532094021"
Sep 09 07:15:41.394911 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:41.394880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:41.400042 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:41.400018    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:42.294232 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:42.294200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:42.295066 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:42.295045    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:15:45.205580 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.205545    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"]
Sep 09 07:15:45.209399 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.209377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:45.211800 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.211781    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-083e4a\""
Sep 09 07:15:45.221463 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.221440    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"]
Sep 09 07:15:45.239642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.239613    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"]
Sep 09 07:15:45.243787 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.243772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:45.246298 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.246282    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-083e4a-dockercfg-vfvlg\""
Sep 09 07:15:45.253207 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.253183    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"]
Sep 09 07:15:45.306178 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.306145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc5h7\" (UniqueName: \"kubernetes.io/projected/3e7484ed-4695-4cf7-94c4-659378e8bccc-kube-api-access-pc5h7\") pod \"payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6\" (UID: \"3e7484ed-4695-4cf7-94c4-659378e8bccc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:45.306178 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.306179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7484ed-4695-4cf7-94c4-659378e8bccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6\" (UID: \"3e7484ed-4695-4cf7-94c4-659378e8bccc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:45.407460 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.407422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f5dcd88-1336-41fc-b816-e4b2e02416a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs\" (UID: \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:45.407625 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.407509    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xcw\" (UniqueName: \"kubernetes.io/projected/1f5dcd88-1336-41fc-b816-e4b2e02416a8-kube-api-access-h6xcw\") pod \"payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs\" (UID: \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:45.407625 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.407541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc5h7\" (UniqueName: \"kubernetes.io/projected/3e7484ed-4695-4cf7-94c4-659378e8bccc-kube-api-access-pc5h7\") pod \"payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6\" (UID: \"3e7484ed-4695-4cf7-94c4-659378e8bccc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:45.407625 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.407566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7484ed-4695-4cf7-94c4-659378e8bccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6\" (UID: \"3e7484ed-4695-4cf7-94c4-659378e8bccc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:45.408242 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.408222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7484ed-4695-4cf7-94c4-659378e8bccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6\" (UID: \"3e7484ed-4695-4cf7-94c4-659378e8bccc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:45.416434 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.416405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc5h7\" (UniqueName: \"kubernetes.io/projected/3e7484ed-4695-4cf7-94c4-659378e8bccc-kube-api-access-pc5h7\") pod \"payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6\" (UID: \"3e7484ed-4695-4cf7-94c4-659378e8bccc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:45.508576 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.508495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f5dcd88-1336-41fc-b816-e4b2e02416a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs\" (UID: \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:45.508576 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.508570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xcw\" (UniqueName: \"kubernetes.io/projected/1f5dcd88-1336-41fc-b816-e4b2e02416a8-kube-api-access-h6xcw\") pod \"payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs\" (UID: \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:45.509215 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.509195    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f5dcd88-1336-41fc-b816-e4b2e02416a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs\" (UID: \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:45.516212 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.516190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xcw\" (UniqueName: \"kubernetes.io/projected/1f5dcd88-1336-41fc-b816-e4b2e02416a8-kube-api-access-h6xcw\") pod \"payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs\" (UID: \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:45.519968 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.519944    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:45.554332 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.554299    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:45.798985 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.798960    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"]
Sep 09 07:15:45.800970 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:45.800944    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7484ed_4695_4cf7_94c4_659378e8bccc.slice/crio-761f427429ce4c76d88d81d7a29d7e9b8017fd5ab5e8da2815b58179f36ee604 WatchSource:0}: Error finding container 761f427429ce4c76d88d81d7a29d7e9b8017fd5ab5e8da2815b58179f36ee604: Status 404 returned error can't find the container with id 761f427429ce4c76d88d81d7a29d7e9b8017fd5ab5e8da2815b58179f36ee604
Sep 09 07:15:45.844083 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:45.844060    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"]
Sep 09 07:15:45.845681 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:45.845656    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5dcd88_1336_41fc_b816_e4b2e02416a8.slice/crio-b13ec747273da98f024f21a397c9a1c35240dd4fb1432d89177a6f13d688814c WatchSource:0}: Error finding container b13ec747273da98f024f21a397c9a1c35240dd4fb1432d89177a6f13d688814c: Status 404 returned error can't find the container with id b13ec747273da98f024f21a397c9a1c35240dd4fb1432d89177a6f13d688814c
Sep 09 07:15:46.350724 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.350683    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs" event={"ID":"1f5dcd88-1336-41fc-b816-e4b2e02416a8","Type":"ContainerStarted","Data":"6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429"}
Sep 09 07:15:46.351242 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.350754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs" event={"ID":"1f5dcd88-1336-41fc-b816-e4b2e02416a8","Type":"ContainerStarted","Data":"b13ec747273da98f024f21a397c9a1c35240dd4fb1432d89177a6f13d688814c"}
Sep 09 07:15:46.351242 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.350912    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:46.352716 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.352693    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6" event={"ID":"3e7484ed-4695-4cf7-94c4-659378e8bccc","Type":"ContainerStarted","Data":"8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49"}
Sep 09 07:15:46.352838 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.352739    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6" event={"ID":"3e7484ed-4695-4cf7-94c4-659378e8bccc","Type":"ContainerStarted","Data":"761f427429ce4c76d88d81d7a29d7e9b8017fd5ab5e8da2815b58179f36ee604"}
Sep 09 07:15:46.352838 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.352770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:46.371195 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.371137    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs" podStartSLOduration=1.371118157 podStartE2EDuration="1.371118157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:46.370944586 +0000 UTC m=+1469.591185700" watchObservedRunningTime="2026-09-09 07:15:46.371118157 +0000 UTC m=+1469.591359265"
Sep 09 07:15:46.379906 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.379808    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"]
Sep 09 07:15:46.380081 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.380027    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw" podUID="a2f6282e-da06-464c-af01-376206ba6262" containerName="payload-processing" containerID="cri-o://ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7" gracePeriod=30
Sep 09 07:15:46.392826 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.392793    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"]
Sep 09 07:15:46.393418 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.393388    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4" podUID="0c364500-f8ca-4f97-ad9f-0b3843d00dd8" containerName="payload-pre-processing" containerID="cri-o://a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169" gracePeriod=30
Sep 09 07:15:46.399608 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.399559    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6" podStartSLOduration=1.399542464 podStartE2EDuration="1.399542464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:46.396255862 +0000 UTC m=+1469.616496974" watchObservedRunningTime="2026-09-09 07:15:46.399542464 +0000 UTC m=+1469.619783577"
Sep 09 07:15:46.560039 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.559996    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"]
Sep 09 07:15:46.567129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.565288    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8398c-h47tw"]
Sep 09 07:15:46.649788 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.649760    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:15:46.699335 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.699307    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:15:46.822900 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.822868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9ltt\" (UniqueName: \"kubernetes.io/projected/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-kube-api-access-g9ltt\") pod \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\" (UID: \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\") "
Sep 09 07:15:46.823129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.822906    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46gt9\" (UniqueName: \"kubernetes.io/projected/a2f6282e-da06-464c-af01-376206ba6262-kube-api-access-46gt9\") pod \"a2f6282e-da06-464c-af01-376206ba6262\" (UID: \"a2f6282e-da06-464c-af01-376206ba6262\") "
Sep 09 07:15:46.823129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.822937    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-plugins-config-volume\") pod \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\" (UID: \"0c364500-f8ca-4f97-ad9f-0b3843d00dd8\") "
Sep 09 07:15:46.823129 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.822966    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f6282e-da06-464c-af01-376206ba6262-plugins-config-volume\") pod \"a2f6282e-da06-464c-af01-376206ba6262\" (UID: \"a2f6282e-da06-464c-af01-376206ba6262\") "
Sep 09 07:15:46.823392 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.823361    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c364500-f8ca-4f97-ad9f-0b3843d00dd8" (UID: "0c364500-f8ca-4f97-ad9f-0b3843d00dd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:15:46.823470 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.823391    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f6282e-da06-464c-af01-376206ba6262-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2f6282e-da06-464c-af01-376206ba6262" (UID: "a2f6282e-da06-464c-af01-376206ba6262"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:15:46.825048 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.825027    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f6282e-da06-464c-af01-376206ba6262-kube-api-access-46gt9" (OuterVolumeSpecName: "kube-api-access-46gt9") pod "a2f6282e-da06-464c-af01-376206ba6262" (UID: "a2f6282e-da06-464c-af01-376206ba6262"). InnerVolumeSpecName "kube-api-access-46gt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:15:46.825150 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.825053    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-kube-api-access-g9ltt" (OuterVolumeSpecName: "kube-api-access-g9ltt") pod "0c364500-f8ca-4f97-ad9f-0b3843d00dd8" (UID: "0c364500-f8ca-4f97-ad9f-0b3843d00dd8"). InnerVolumeSpecName "kube-api-access-g9ltt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:15:46.924298 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.924271    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9ltt\" (UniqueName: \"kubernetes.io/projected/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-kube-api-access-g9ltt\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:46.924298 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.924295    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46gt9\" (UniqueName: \"kubernetes.io/projected/a2f6282e-da06-464c-af01-376206ba6262-kube-api-access-46gt9\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:46.924468 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.924306    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c364500-f8ca-4f97-ad9f-0b3843d00dd8-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:46.924468 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:46.924315    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f6282e-da06-464c-af01-376206ba6262-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:47.231902 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.231819    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f01d809c-8928-4177-bae2-006913f85dfb" path="/var/lib/kubelet/pods/f01d809c-8928-4177-bae2-006913f85dfb/volumes"
Sep 09 07:15:47.361428 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.361390    2562 generic.go:356] "Generic (PLEG): container finished" podID="a2f6282e-da06-464c-af01-376206ba6262" containerID="ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7" exitCode=0
Sep 09 07:15:47.361911 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.361440    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"
Sep 09 07:15:47.361911 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.361484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw" event={"ID":"a2f6282e-da06-464c-af01-376206ba6262","Type":"ContainerDied","Data":"ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7"}
Sep 09 07:15:47.361911 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.361543    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw" event={"ID":"a2f6282e-da06-464c-af01-376206ba6262","Type":"ContainerDied","Data":"05d89bda9a53e9c3dc0289b52beb62d9064c019bdc43781e68ab2889fe9673f6"}
Sep 09 07:15:47.361911 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.361574    2562 scope.go:117] "RemoveContainer" containerID="ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7"
Sep 09 07:15:47.362982 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.362961    2562 generic.go:356] "Generic (PLEG): container finished" podID="0c364500-f8ca-4f97-ad9f-0b3843d00dd8" containerID="a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169" exitCode=0
Sep 09 07:15:47.363088 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.363007    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"
Sep 09 07:15:47.363088 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.363061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4" event={"ID":"0c364500-f8ca-4f97-ad9f-0b3843d00dd8","Type":"ContainerDied","Data":"a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169"}
Sep 09 07:15:47.363205 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.363144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4" event={"ID":"0c364500-f8ca-4f97-ad9f-0b3843d00dd8","Type":"ContainerDied","Data":"32b7752416cbc55a5b8e7a43095d25135e82d39dad320feecf30f648059ed08e"}
Sep 09 07:15:47.372208 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.372188    2562 scope.go:117] "RemoveContainer" containerID="ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7"
Sep 09 07:15:47.372506 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:15:47.372482    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7\": container with ID starting with ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7 not found: ID does not exist" containerID="ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7"
Sep 09 07:15:47.372612 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.372512    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7"} err="failed to get container status \"ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7\": rpc error: code = NotFound desc = could not find container \"ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7\": container with ID starting with ba9411d033a678225c00b63d6c7a3e55c01420f3af59dc3b0814c4ed8d62aff7 not found: ID does not exist"
Sep 09 07:15:47.372612 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.372529    2562 scope.go:117] "RemoveContainer" containerID="a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169"
Sep 09 07:15:47.381881 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.381863    2562 scope.go:117] "RemoveContainer" containerID="a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169"
Sep 09 07:15:47.382167 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:15:47.382140    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169\": container with ID starting with a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169 not found: ID does not exist" containerID="a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169"
Sep 09 07:15:47.382259 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.382177    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169"} err="failed to get container status \"a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169\": rpc error: code = NotFound desc = could not find container \"a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169\": container with ID starting with a2f610652ab2022a7f0b7a9c4b6ccdb265c279913c8f863b577f426847911169 not found: ID does not exist"
Sep 09 07:15:47.382695 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.382678    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"]
Sep 09 07:15:47.388306 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.388284    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8398c-6c4fcbf746-vxmt4"]
Sep 09 07:15:47.398182 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.398160    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"]
Sep 09 07:15:47.402455 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:47.402434    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8398c-59dbf99f9f-b4ctw"]
Sep 09 07:15:49.230613 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:49.230579    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c364500-f8ca-4f97-ad9f-0b3843d00dd8" path="/var/lib/kubelet/pods/0c364500-f8ca-4f97-ad9f-0b3843d00dd8/volumes"
Sep 09 07:15:49.230986 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:49.230943    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f6282e-da06-464c-af01-376206ba6262" path="/var/lib/kubelet/pods/a2f6282e-da06-464c-af01-376206ba6262/volumes"
Sep 09 07:15:52.276606 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:52.276562    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:15:53.997575 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:53.997530    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"]
Sep 09 07:15:53.998089 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:53.998073    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f6282e-da06-464c-af01-376206ba6262" containerName="payload-processing"
Sep 09 07:15:53.998182 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:53.998094    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f6282e-da06-464c-af01-376206ba6262" containerName="payload-processing"
Sep 09 07:15:53.998182 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:53.998167    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c364500-f8ca-4f97-ad9f-0b3843d00dd8" containerName="payload-pre-processing"
Sep 09 07:15:53.998182 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:53.998177    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c364500-f8ca-4f97-ad9f-0b3843d00dd8" containerName="payload-pre-processing"
Sep 09 07:15:53.998352 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:53.998273    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c364500-f8ca-4f97-ad9f-0b3843d00dd8" containerName="payload-pre-processing"
Sep 09 07:15:53.998352 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:53.998286    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f6282e-da06-464c-af01-376206ba6262" containerName="payload-processing"
Sep 09 07:15:54.002521 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.002498    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"
Sep 09 07:15:54.005347 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.005082    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:15:54.005347 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.005342    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:15:54.008595 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.008574    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"]
Sep 09 07:15:54.089972 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.089937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe765fc0-534a-4aba-8f2d-e963aca667d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv\" (UID: \"fe765fc0-534a-4aba-8f2d-e963aca667d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"
Sep 09 07:15:54.191517 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.191475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe765fc0-534a-4aba-8f2d-e963aca667d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv\" (UID: \"fe765fc0-534a-4aba-8f2d-e963aca667d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"
Sep 09 07:15:54.192315 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.192291    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe765fc0-534a-4aba-8f2d-e963aca667d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv\" (UID: \"fe765fc0-534a-4aba-8f2d-e963aca667d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"
Sep 09 07:15:54.314863 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.314772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"
Sep 09 07:15:54.617529 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:54.617499    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"]
Sep 09 07:15:54.619114 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:15:54.619074    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe765fc0_534a_4aba_8f2d_e963aca667d7.slice/crio-c7756e90032d9858b21883004c8f960c6ccef32f01c4b34d99a81915e50a97f5 WatchSource:0}: Error finding container c7756e90032d9858b21883004c8f960c6ccef32f01c4b34d99a81915e50a97f5: Status 404 returned error can't find the container with id c7756e90032d9858b21883004c8f960c6ccef32f01c4b34d99a81915e50a97f5
Sep 09 07:15:55.426662 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:55.426621    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv" event={"ID":"fe765fc0-534a-4aba-8f2d-e963aca667d7","Type":"ContainerStarted","Data":"36ca5ba98e675d09c2722f590a8311654919f69663c48ecf4ef5d97a8091b690"}
Sep 09 07:15:55.427084 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:55.426675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv" event={"ID":"fe765fc0-534a-4aba-8f2d-e963aca667d7","Type":"ContainerStarted","Data":"c7756e90032d9858b21883004c8f960c6ccef32f01c4b34d99a81915e50a97f5"}
Sep 09 07:15:55.442797 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:55.442744    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv" podStartSLOduration=2.44273093 podStartE2EDuration="2.44273093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:15:55.440999666 +0000 UTC m=+1478.661240789" watchObservedRunningTime="2026-09-09 07:15:55.44273093 +0000 UTC m=+1478.662972043"
Sep 09 07:15:56.437697 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:56.437660    2562 generic.go:356] "Generic (PLEG): container finished" podID="fe765fc0-534a-4aba-8f2d-e963aca667d7" containerID="36ca5ba98e675d09c2722f590a8311654919f69663c48ecf4ef5d97a8091b690" exitCode=0
Sep 09 07:15:56.438164 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:56.437746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv" event={"ID":"fe765fc0-534a-4aba-8f2d-e963aca667d7","Type":"ContainerDied","Data":"36ca5ba98e675d09c2722f590a8311654919f69663c48ecf4ef5d97a8091b690"}
Sep 09 07:15:57.366267 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:57.366230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:15:57.366587 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:57.366562    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:15:57.619715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:57.619654    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"
Sep 09 07:15:57.732746 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:57.732711    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe765fc0-534a-4aba-8f2d-e963aca667d7-maas-api-service-ca\") pod \"fe765fc0-534a-4aba-8f2d-e963aca667d7\" (UID: \"fe765fc0-534a-4aba-8f2d-e963aca667d7\") "
Sep 09 07:15:57.733167 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:57.733139    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe765fc0-534a-4aba-8f2d-e963aca667d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe765fc0-534a-4aba-8f2d-e963aca667d7" (UID: "fe765fc0-534a-4aba-8f2d-e963aca667d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:15:57.834303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:57.834269    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe765fc0-534a-4aba-8f2d-e963aca667d7-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:15:58.454111 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:58.454058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv" event={"ID":"fe765fc0-534a-4aba-8f2d-e963aca667d7","Type":"ContainerDied","Data":"c7756e90032d9858b21883004c8f960c6ccef32f01c4b34d99a81915e50a97f5"}
Sep 09 07:15:58.454111 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:58.454077    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"
Sep 09 07:15:58.454373 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:15:58.454129    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7756e90032d9858b21883004c8f960c6ccef32f01c4b34d99a81915e50a97f5"
Sep 09 07:16:02.483221 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:02.483184    2562 generic.go:356] "Generic (PLEG): container finished" podID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerID="dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a" exitCode=0
Sep 09 07:16:02.483771 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:02.483262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" event={"ID":"7e58b0b8-ff48-46d1-b7e5-fa7b63295609","Type":"ContainerDied","Data":"dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a"}
Sep 09 07:16:02.483844 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:02.483787    2562 scope.go:117] "RemoveContainer" containerID="dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a"
Sep 09 07:16:02.485021 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:02.484989    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:16:02.485132 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:02.485058    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:16:02.485132 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:02.485093    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 09 07:16:03.495438 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:03.495404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" event={"ID":"7e58b0b8-ff48-46d1-b7e5-fa7b63295609","Type":"ContainerStarted","Data":"4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1"}
Sep 09 07:16:04.394534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:04.394499    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:16:04.394778 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:04.394753    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 09 07:16:04.394827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:04.394806    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 09 07:16:04.785390 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:04.785350    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"]
Sep 09 07:16:04.791460 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:04.791427    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1128ac-x4chv"]
Sep 09 07:16:05.231091 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:05.231057    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe765fc0-534a-4aba-8f2d-e963aca667d7" path="/var/lib/kubelet/pods/fe765fc0-534a-4aba-8f2d-e963aca667d7/volumes"
Sep 09 07:16:05.394795 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:05.394751    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 09 07:16:05.394975 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:05.394818    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 09 07:16:06.394348 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:06.394315    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 09 07:16:06.394775 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:06.394374    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 09 07:16:07.395013 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:07.394977    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 09 07:16:07.395448 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:07.395034    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 09 07:16:08.395348 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:08.395311    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 09 07:16:08.395803 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:08.395369    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 09 07:16:09.398558 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:09.398529    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:16:09.398930 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:09.398872    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:16:09.399536 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:09.399519    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:16:09.476075 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:09.476044    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:16:11.404159 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.404117    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"]
Sep 09 07:16:11.404645 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.404398    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy" containerID="cri-o://6040f94d8c76038a9786e296c258f235cac8b0c44fa5bba6f8cc0ea53831aa7c" gracePeriod=30
Sep 09 07:16:11.787481 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.787447    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"]
Sep 09 07:16:11.787827 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.787815    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe765fc0-534a-4aba-8f2d-e963aca667d7" containerName="revoke-keys"
Sep 09 07:16:11.787877 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.787829    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe765fc0-534a-4aba-8f2d-e963aca667d7" containerName="revoke-keys"
Sep 09 07:16:11.787924 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.787915    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe765fc0-534a-4aba-8f2d-e963aca667d7" containerName="revoke-keys"
Sep 09 07:16:11.790726 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.790706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"
Sep 09 07:16:11.793292 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.793268    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:16:11.793399 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.793378    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:16:11.799351 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.799329    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"]
Sep 09 07:16:11.876511 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.876482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6\" (UID: \"f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"
Sep 09 07:16:11.977308 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.977275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6\" (UID: \"f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"
Sep 09 07:16:11.977907 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:11.977885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6\" (UID: \"f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"
Sep 09 07:16:12.101779 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:12.101690    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"
Sep 09 07:16:12.373289 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:12.373256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"]
Sep 09 07:16:12.374719 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:16:12.374692    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bb5eb6_3f9a_4fd5_8923_780e33b59fc1.slice/crio-99aef3e3e8bebc495dd8d984e24222e069442de98c5ba5b3e6d5c72f1c7f3ca2 WatchSource:0}: Error finding container 99aef3e3e8bebc495dd8d984e24222e069442de98c5ba5b3e6d5c72f1c7f3ca2: Status 404 returned error can't find the container with id 99aef3e3e8bebc495dd8d984e24222e069442de98c5ba5b3e6d5c72f1c7f3ca2
Sep 09 07:16:12.551677 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:12.551640    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6" event={"ID":"f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1","Type":"ContainerStarted","Data":"99aef3e3e8bebc495dd8d984e24222e069442de98c5ba5b3e6d5c72f1c7f3ca2"}
Sep 09 07:16:13.559680 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:13.559650    2562 generic.go:356] "Generic (PLEG): container finished" podID="f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1" containerID="b626c08ac1a1750e24406d23eb64c7ff105c9aaad16e4abc7abeeb20e1d98f0a" exitCode=0
Sep 09 07:16:13.560078 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:13.559713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6" event={"ID":"f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1","Type":"ContainerDied","Data":"b626c08ac1a1750e24406d23eb64c7ff105c9aaad16e4abc7abeeb20e1d98f0a"}
Sep 09 07:16:13.651914 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:13.651880    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 07:16:13.652081 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:13.651934    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 07:16:14.129872 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:14.129841    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs_1f5dcd88-1336-41fc-b816-e4b2e02416a8/payload-processing/0.log"
Sep 09 07:16:14.698459 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:14.698431    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"
Sep 09 07:16:14.809098 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:14.809065    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1-maas-api-service-ca\") pod \"f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1\" (UID: \"f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1\") "
Sep 09 07:16:14.809496 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:14.809471    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1" (UID: "f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:16:14.910346 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:14.910258    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:15.572858 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:15.572775    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"
Sep 09 07:16:15.572858 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:15.572803    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6" event={"ID":"f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1","Type":"ContainerDied","Data":"99aef3e3e8bebc495dd8d984e24222e069442de98c5ba5b3e6d5c72f1c7f3ca2"}
Sep 09 07:16:15.572858 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:15.572851    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99aef3e3e8bebc495dd8d984e24222e069442de98c5ba5b3e6d5c72f1c7f3ca2"
Sep 09 07:16:16.582224 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.582186    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerID="6040f94d8c76038a9786e296c258f235cac8b0c44fa5bba6f8cc0ea53831aa7c" exitCode=0
Sep 09 07:16:16.582665 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.582263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" event={"ID":"7fb91a24-9fab-4fc7-a613-2a901881c5dc","Type":"ContainerDied","Data":"6040f94d8c76038a9786e296c258f235cac8b0c44fa5bba6f8cc0ea53831aa7c"}
Sep 09 07:16:16.582665 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.582336    2562 scope.go:117] "RemoveContainer" containerID="5cfd3ab1f49c92415dd370eeb745555e049f970ae1fb7b57d2b0cf844355a7b2"
Sep 09 07:16:16.651366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.651344    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:16:16.727820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.727736    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-token\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.727820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.727783    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-podinfo\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.727820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.727818    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-envoy\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.728123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.727864    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-credential-socket\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.728123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.727905    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-certs\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.728123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.727925    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-socket\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.728123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.727953    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istiod-ca-cert\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.728123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.727976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-data\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.728123 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728029    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwq5r\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-kube-api-access-dwq5r\") pod \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\" (UID: \"7fb91a24-9fab-4fc7-a613-2a901881c5dc\") "
Sep 09 07:16:16.728473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728227    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:16.728473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728270    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:16.728473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728287    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:16.728473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728413    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:16.728473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728431    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:16.728473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728446    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:16.728473 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728465    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:16.728846 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.728639    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:16:16.730022 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.730000    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:16:16.730154 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.730061    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:16:16.730154 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.730141    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:16.730253 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.730237    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-kube-api-access-dwq5r" (OuterVolumeSpecName: "kube-api-access-dwq5r") pod "7fb91a24-9fab-4fc7-a613-2a901881c5dc" (UID: "7fb91a24-9fab-4fc7-a613-2a901881c5dc"). InnerVolumeSpecName "kube-api-access-dwq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:16:16.829179 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.829143    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:16.829179 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.829174    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:16.829179 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.829182    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:16.829416 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.829192    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:16.829416 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.829201    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb91a24-9fab-4fc7-a613-2a901881c5dc-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:16.829416 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:16.829210    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwq5r\" (UniqueName: \"kubernetes.io/projected/7fb91a24-9fab-4fc7-a613-2a901881c5dc-kube-api-access-dwq5r\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:17.284564 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:17.284529    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:16:17.290610 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:17.290589    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:16:17.501818 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:17.501732    2562 scope.go:117] "RemoveContainer" containerID="6040f94d8c76038a9786e296c258f235cac8b0c44fa5bba6f8cc0ea53831aa7c"
Sep 09 07:16:17.588429 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:17.588397    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn" event={"ID":"7fb91a24-9fab-4fc7-a613-2a901881c5dc","Type":"ContainerDied","Data":"22c52070c7bf151667f26b844f6893323d316b5b943167dcd0b1b6136059007d"}
Sep 09 07:16:17.588754 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:17.588428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"
Sep 09 07:16:17.621956 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:17.621932    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"]
Sep 09 07:16:17.625916 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:17.625894    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1128ac-openshift-default-b975bdf66-pz8mn"]
Sep 09 07:16:19.230253 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:19.230220    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" path="/var/lib/kubelet/pods/7fb91a24-9fab-4fc7-a613-2a901881c5dc/volumes"
Sep 09 07:16:22.611798 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.611759    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"]
Sep 09 07:16:22.615078 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.612674    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5" podUID="5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd" containerName="payload-pre-processing" containerID="cri-o://b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555" gracePeriod=30
Sep 09 07:16:22.786879 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.786848    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"]
Sep 09 07:16:22.790135 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.790085    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-1d8843-5z9m6"]
Sep 09 07:16:22.875097 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.875040    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:16:22.985316 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.985283    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfqsd\" (UniqueName: \"kubernetes.io/projected/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-kube-api-access-sfqsd\") pod \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\" (UID: \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\") "
Sep 09 07:16:22.985518 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.985433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-plugins-config-volume\") pod \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\" (UID: \"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd\") "
Sep 09 07:16:22.985806 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.985782    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd" (UID: "5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:16:22.987313 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:22.987288    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-kube-api-access-sfqsd" (OuterVolumeSpecName: "kube-api-access-sfqsd") pod "5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd" (UID: "5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd"). InnerVolumeSpecName "kube-api-access-sfqsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:16:23.086609 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.086572    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:23.086609 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.086604    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfqsd\" (UniqueName: \"kubernetes.io/projected/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd-kube-api-access-sfqsd\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:23.229975 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.229929    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1" path="/var/lib/kubelet/pods/f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1/volumes"
Sep 09 07:16:23.632615 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.632532    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd" containerID="b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555" exitCode=0
Sep 09 07:16:23.632615 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.632576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5" event={"ID":"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd","Type":"ContainerDied","Data":"b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555"}
Sep 09 07:16:23.632615 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.632592    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"
Sep 09 07:16:23.633178 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.632636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5" event={"ID":"5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd","Type":"ContainerDied","Data":"266d9e5d718815053e20533f97a5ad7acc45ae52791af12adeeac69e26b48747"}
Sep 09 07:16:23.633178 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.632664    2562 scope.go:117] "RemoveContainer" containerID="b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555"
Sep 09 07:16:23.642355 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.642335    2562 scope.go:117] "RemoveContainer" containerID="b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555"
Sep 09 07:16:23.642637 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:16:23.642614    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555\": container with ID starting with b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555 not found: ID does not exist" containerID="b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555"
Sep 09 07:16:23.642729 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.642649    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555"} err="failed to get container status \"b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555\": rpc error: code = NotFound desc = could not find container \"b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555\": container with ID starting with b7b04fc2027c3ed3343f853ed7248e371b6ba10b4045b422c13e0068263a4555 not found: ID does not exist"
Sep 09 07:16:23.654939 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.654912    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"]
Sep 09 07:16:23.658559 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:23.658536    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d8843-669d66bc46-p8tr5"]
Sep 09 07:16:25.231121 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:25.231083    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd" path="/var/lib/kubelet/pods/5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd/volumes"
Sep 09 07:16:26.868570 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:26.868537    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:16:29.225027 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:29.224984    2562 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" secret="" err="secret \"e2e-shared-b-w2-1d8843-openshift-default-dockercfg-wb7db\" not found"
Sep 09 07:16:29.237577 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:29.237545    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"]
Sep 09 07:16:29.674842 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:29.674799    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy" containerID="cri-o://8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1" gracePeriod=30
Sep 09 07:16:29.993661 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:29.993577    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 07:16:29.993661 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:29.993647    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 07:16:34.927156 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.927129    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:16:34.994744 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994674    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2jj5\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-kube-api-access-s2jj5\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.994744 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-credential-socket\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.994744 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-certs\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.995094 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994778    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-istio-token\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.995094 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994796    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-data\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.995094 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994817    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732324b1-73bf-4d7c-9317-648b663c5b22-istio-podinfo\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.995094 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994855    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-envoy\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.995094 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994909    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732324b1-73bf-4d7c-9317-648b663c5b22-istiod-ca-cert\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.995094 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994936    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-socket\") pod \"732324b1-73bf-4d7c-9317-648b663c5b22\" (UID: \"732324b1-73bf-4d7c-9317-648b663c5b22\") "
Sep 09 07:16:34.995094 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.994979    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:34.995481 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.995176    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-data" (OuterVolumeSpecName: "istio-data") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:34.995481 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.995286    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:34.995481 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.995308    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:34.995481 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.995416    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:34.995481 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.995445    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:34.995695 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.995636    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732324b1-73bf-4d7c-9317-648b663c5b22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:16:34.997124 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.997073    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:16:34.997217 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.997123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/732324b1-73bf-4d7c-9317-648b663c5b22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:16:34.997369 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.997350    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-istio-token" (OuterVolumeSpecName: "istio-token") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:16:34.997444 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:34.997415    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-kube-api-access-s2jj5" (OuterVolumeSpecName: "kube-api-access-s2jj5") pod "732324b1-73bf-4d7c-9317-648b663c5b22" (UID: "732324b1-73bf-4d7c-9317-648b663c5b22"). InnerVolumeSpecName "kube-api-access-s2jj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:16:35.095715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.095689    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:35.095715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.095711    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732324b1-73bf-4d7c-9317-648b663c5b22-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:35.095715 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.095720    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:35.095908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.095728    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732324b1-73bf-4d7c-9317-648b663c5b22-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:35.095908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.095737    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:35.095908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.095745    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2jj5\" (UniqueName: \"kubernetes.io/projected/732324b1-73bf-4d7c-9317-648b663c5b22-kube-api-access-s2jj5\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:35.095908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.095754    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732324b1-73bf-4d7c-9317-648b663c5b22-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:16:35.712472 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.712437    2562 generic.go:356] "Generic (PLEG): container finished" podID="732324b1-73bf-4d7c-9317-648b663c5b22" containerID="8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1" exitCode=0
Sep 09 07:16:35.712654 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.712485    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"
Sep 09 07:16:35.712654 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.712517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" event={"ID":"732324b1-73bf-4d7c-9317-648b663c5b22","Type":"ContainerDied","Data":"8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1"}
Sep 09 07:16:35.712654 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.712571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz" event={"ID":"732324b1-73bf-4d7c-9317-648b663c5b22","Type":"ContainerDied","Data":"61f00ee4074ab33171e27e7e5c3ceeb282a9217199230ded07d40e855e43b7e1"}
Sep 09 07:16:35.712654 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.712594    2562 scope.go:117] "RemoveContainer" containerID="8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1"
Sep 09 07:16:35.722261 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.722242    2562 scope.go:117] "RemoveContainer" containerID="8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319"
Sep 09 07:16:35.730985 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.730957    2562 scope.go:117] "RemoveContainer" containerID="8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1"
Sep 09 07:16:35.731279 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:16:35.731260    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1\": container with ID starting with 8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1 not found: ID does not exist" containerID="8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1"
Sep 09 07:16:35.731343 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.731288    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1"} err="failed to get container status \"8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1\": rpc error: code = NotFound desc = could not find container \"8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1\": container with ID starting with 8a9f191aa4c522cd204b68cd1c864b549056ff3d7e0ba6fa6924379a19bcf2d1 not found: ID does not exist"
Sep 09 07:16:35.731343 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.731305    2562 scope.go:117] "RemoveContainer" containerID="8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319"
Sep 09 07:16:35.731442 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.731401    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"]
Sep 09 07:16:35.731589 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:16:35.731572    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319\": container with ID starting with 8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319 not found: ID does not exist" containerID="8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319"
Sep 09 07:16:35.731629 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.731596    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319"} err="failed to get container status \"8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319\": rpc error: code = NotFound desc = could not find container \"8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319\": container with ID starting with 8b824864377cdab6f0261cebb45336cfc9799bbe1f146b2607e5fcfa8071d319 not found: ID does not exist"
Sep 09 07:16:35.734875 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:35.734855    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1d8843-openshift-default-79c596c4bd-r5kpz"]
Sep 09 07:16:37.230572 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:37.230539    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" path="/var/lib/kubelet/pods/732324b1-73bf-4d7c-9317-648b663c5b22/volumes"
Sep 09 07:16:38.962224 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:38.962187    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:16:50.863516 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:50.863477    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:16:57.891067 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:16:57.891038    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs_1f5dcd88-1336-41fc-b816-e4b2e02416a8/payload-processing/0.log"
Sep 09 07:17:04.001797 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.001759    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"]
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002125    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002138    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002149    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002155    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002171    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd" containerName="payload-pre-processing"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002179    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd" containerName="payload-pre-processing"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002188    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002193    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002201    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1" containerName="revoke-keys"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002206    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1" containerName="revoke-keys"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002278    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6bb5eb6-3f9a-4fd5-8923-780e33b59fc1" containerName="revoke-keys"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002288    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy"
Sep 09 07:17:04.002288 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002294    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy"
Sep 09 07:17:04.002703 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002302    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e01c5d7-4456-41d8-a9f1-cf8cd40cdcbd" containerName="payload-pre-processing"
Sep 09 07:17:04.002703 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002309    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy"
Sep 09 07:17:04.002703 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.002314    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fb91a24-9fab-4fc7-a613-2a901881c5dc" containerName="istio-proxy"
Sep 09 07:17:04.006196 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.006163    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.011661 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.011618    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-36a83e\""
Sep 09 07:17:04.011959 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.011924    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-36a83e-dockercfg-zm29q\""
Sep 09 07:17:04.015065 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.015035    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"]
Sep 09 07:17:04.015581 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.015556    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy"
Sep 09 07:17:04.015581 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.015582    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="732324b1-73bf-4d7c-9317-648b663c5b22" containerName="istio-proxy"
Sep 09 07:17:04.018760 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.018741    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.042542 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.042518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"]
Sep 09 07:17:04.066267 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.066239    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"]
Sep 09 07:17:04.162832 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.162794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g\" (UID: \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.163038 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.162844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbdm\" (UniqueName: \"kubernetes.io/projected/5d3f7924-4384-41c2-8cfd-a544101159dc-kube-api-access-krbdm\") pod \"payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d\" (UID: \"5d3f7924-4384-41c2-8cfd-a544101159dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.163038 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.162966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3f7924-4384-41c2-8cfd-a544101159dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d\" (UID: \"5d3f7924-4384-41c2-8cfd-a544101159dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.163194 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.163039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwkh9\" (UniqueName: \"kubernetes.io/projected/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-kube-api-access-wwkh9\") pod \"payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g\" (UID: \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.264520 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.264425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g\" (UID: \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.264520 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.264472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbdm\" (UniqueName: \"kubernetes.io/projected/5d3f7924-4384-41c2-8cfd-a544101159dc-kube-api-access-krbdm\") pod \"payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d\" (UID: \"5d3f7924-4384-41c2-8cfd-a544101159dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.264733 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.264550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3f7924-4384-41c2-8cfd-a544101159dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d\" (UID: \"5d3f7924-4384-41c2-8cfd-a544101159dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.264733 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.264608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwkh9\" (UniqueName: \"kubernetes.io/projected/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-kube-api-access-wwkh9\") pod \"payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g\" (UID: \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.265257 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.265236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g\" (UID: \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.265475 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.265450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3f7924-4384-41c2-8cfd-a544101159dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d\" (UID: \"5d3f7924-4384-41c2-8cfd-a544101159dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.272814 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.272786    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbdm\" (UniqueName: \"kubernetes.io/projected/5d3f7924-4384-41c2-8cfd-a544101159dc-kube-api-access-krbdm\") pod \"payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d\" (UID: \"5d3f7924-4384-41c2-8cfd-a544101159dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.273161 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.273140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwkh9\" (UniqueName: \"kubernetes.io/projected/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-kube-api-access-wwkh9\") pod \"payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g\" (UID: \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.318999 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.318970    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.328199 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.328178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.600542 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.600518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"]
Sep 09 07:17:04.603383 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:17:04.603355    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052ca5a3_9ddc_4a37_a460_6d87edf7b7f9.slice/crio-f5d6164d75f684054b23f4711b9a422f9e6447380fa4462222b24b14bc40bd33 WatchSource:0}: Error finding container f5d6164d75f684054b23f4711b9a422f9e6447380fa4462222b24b14bc40bd33: Status 404 returned error can't find the container with id f5d6164d75f684054b23f4711b9a422f9e6447380fa4462222b24b14bc40bd33
Sep 09 07:17:04.604340 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.604315    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"]
Sep 09 07:17:04.605388 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:17:04.605366    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3f7924_4384_41c2_8cfd_a544101159dc.slice/crio-ef76a6129e11ca1b850ebb61e9b61f74ae0316d251496f11cc4c4a524abbd47f WatchSource:0}: Error finding container ef76a6129e11ca1b850ebb61e9b61f74ae0316d251496f11cc4c4a524abbd47f: Status 404 returned error can't find the container with id ef76a6129e11ca1b850ebb61e9b61f74ae0316d251496f11cc4c4a524abbd47f
Sep 09 07:17:04.888485 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.888385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g" event={"ID":"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9","Type":"ContainerStarted","Data":"cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd"}
Sep 09 07:17:04.888485 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.888447    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g" event={"ID":"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9","Type":"ContainerStarted","Data":"f5d6164d75f684054b23f4711b9a422f9e6447380fa4462222b24b14bc40bd33"}
Sep 09 07:17:04.888485 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.888477    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:04.890160 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.890131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d" event={"ID":"5d3f7924-4384-41c2-8cfd-a544101159dc","Type":"ContainerStarted","Data":"5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205"}
Sep 09 07:17:04.890290 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.890190    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:04.890290 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.890216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d" event={"ID":"5d3f7924-4384-41c2-8cfd-a544101159dc","Type":"ContainerStarted","Data":"ef76a6129e11ca1b850ebb61e9b61f74ae0316d251496f11cc4c4a524abbd47f"}
Sep 09 07:17:04.908902 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.908843    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g" podStartSLOduration=1.908825003 podStartE2EDuration="1.908825003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:17:04.905889984 +0000 UTC m=+1548.126131097" watchObservedRunningTime="2026-09-09 07:17:04.908825003 +0000 UTC m=+1548.129066117"
Sep 09 07:17:04.925690 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:04.925644    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d" podStartSLOduration=1.925628658 podStartE2EDuration="1.925628658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:17:04.922006479 +0000 UTC m=+1548.142247592" watchObservedRunningTime="2026-09-09 07:17:04.925628658 +0000 UTC m=+1548.145869771"
Sep 09 07:17:10.369808 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:10.369763    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:17:15.898762 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:15.898731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:15.899196 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:15.898809    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:17.544586 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:17.544555    2562 scope.go:117] "RemoveContainer" containerID="70e71545f953c2461bad29ce731f98966cccf1b2975d11bd0c87eff5e8ddfe5e"
Sep 09 07:17:17.553458 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:17.553438    2562 scope.go:117] "RemoveContainer" containerID="e09db6d46f6025d8c77bf874da4d3f6033172fc699c95a6f8f594d8fab44a5ed"
Sep 09 07:17:27.976864 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:27.976824    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"]
Sep 09 07:17:27.980458 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:27.980442    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"
Sep 09 07:17:27.983434 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:27.983415    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:17:27.983879 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:27.983862    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:17:27.991293 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:27.991265    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"]
Sep 09 07:17:28.078501 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:28.078463    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13d2121-75b7-4982-8507-dd345586a8a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q\" (UID: \"e13d2121-75b7-4982-8507-dd345586a8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"
Sep 09 07:17:28.179603 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:28.179562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13d2121-75b7-4982-8507-dd345586a8a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q\" (UID: \"e13d2121-75b7-4982-8507-dd345586a8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"
Sep 09 07:17:28.180202 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:28.180183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13d2121-75b7-4982-8507-dd345586a8a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q\" (UID: \"e13d2121-75b7-4982-8507-dd345586a8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"
Sep 09 07:17:28.291579 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:28.291476    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"
Sep 09 07:17:28.563535 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:28.563510    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"]
Sep 09 07:17:28.565500 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:17:28.565463    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13d2121_75b7_4982_8507_dd345586a8a6.slice/crio-112d61b4efae3116b2a893e666d4c495bfc4b9c68eaf3d9252f6b4118bc7f659 WatchSource:0}: Error finding container 112d61b4efae3116b2a893e666d4c495bfc4b9c68eaf3d9252f6b4118bc7f659: Status 404 returned error can't find the container with id 112d61b4efae3116b2a893e666d4c495bfc4b9c68eaf3d9252f6b4118bc7f659
Sep 09 07:17:29.058741 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:29.058705    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q" event={"ID":"e13d2121-75b7-4982-8507-dd345586a8a6","Type":"ContainerStarted","Data":"b018302c15c495ef26d4a65c613714ce480312d8e03465db54e2b6fc5d7de2e7"}
Sep 09 07:17:29.059175 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:29.058770    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q" event={"ID":"e13d2121-75b7-4982-8507-dd345586a8a6","Type":"ContainerStarted","Data":"112d61b4efae3116b2a893e666d4c495bfc4b9c68eaf3d9252f6b4118bc7f659"}
Sep 09 07:17:29.077469 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:29.077414    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q" podStartSLOduration=2.077398382 podStartE2EDuration="2.077398382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:17:29.07645008 +0000 UTC m=+1572.296691194" watchObservedRunningTime="2026-09-09 07:17:29.077398382 +0000 UTC m=+1572.297639496"
Sep 09 07:17:30.066665 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:30.066629    2562 generic.go:356] "Generic (PLEG): container finished" podID="e13d2121-75b7-4982-8507-dd345586a8a6" containerID="b018302c15c495ef26d4a65c613714ce480312d8e03465db54e2b6fc5d7de2e7" exitCode=0
Sep 09 07:17:30.067124 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:30.066716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q" event={"ID":"e13d2121-75b7-4982-8507-dd345586a8a6","Type":"ContainerDied","Data":"b018302c15c495ef26d4a65c613714ce480312d8e03465db54e2b6fc5d7de2e7"}
Sep 09 07:17:31.207745 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:31.207719    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"
Sep 09 07:17:31.309200 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:31.309161    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13d2121-75b7-4982-8507-dd345586a8a6-maas-api-service-ca\") pod \"e13d2121-75b7-4982-8507-dd345586a8a6\" (UID: \"e13d2121-75b7-4982-8507-dd345586a8a6\") "
Sep 09 07:17:31.309536 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:31.309512    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13d2121-75b7-4982-8507-dd345586a8a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e13d2121-75b7-4982-8507-dd345586a8a6" (UID: "e13d2121-75b7-4982-8507-dd345586a8a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:17:31.410930 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:31.410827    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13d2121-75b7-4982-8507-dd345586a8a6-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:17:32.080209 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:32.080183    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"
Sep 09 07:17:32.080383 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:32.080229    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q" event={"ID":"e13d2121-75b7-4982-8507-dd345586a8a6","Type":"ContainerDied","Data":"112d61b4efae3116b2a893e666d4c495bfc4b9c68eaf3d9252f6b4118bc7f659"}
Sep 09 07:17:32.080383 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:32.080273    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112d61b4efae3116b2a893e666d4c495bfc4b9c68eaf3d9252f6b4118bc7f659"
Sep 09 07:17:33.991499 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:33.991466    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"]
Sep 09 07:17:33.991988 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:33.991725    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d" podUID="5d3f7924-4384-41c2-8cfd-a544101159dc" containerName="payload-processing" containerID="cri-o://5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205" gracePeriod=30
Sep 09 07:17:33.996935 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:33.996907    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"]
Sep 09 07:17:33.997137 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:33.997114    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g" podUID="052ca5a3-9ddc-4a37-a460-6d87edf7b7f9" containerName="payload-pre-processing" containerID="cri-o://cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd" gracePeriod=30
Sep 09 07:17:34.179905 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.179867    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"]
Sep 09 07:17:34.183973 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.183947    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36a83e-j8n2q"]
Sep 09 07:17:34.244431 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.244357    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:34.275008 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.274983    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:34.336877 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.336842    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krbdm\" (UniqueName: \"kubernetes.io/projected/5d3f7924-4384-41c2-8cfd-a544101159dc-kube-api-access-krbdm\") pod \"5d3f7924-4384-41c2-8cfd-a544101159dc\" (UID: \"5d3f7924-4384-41c2-8cfd-a544101159dc\") "
Sep 09 07:17:34.336877 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.336884    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-plugins-config-volume\") pod \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\" (UID: \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\") "
Sep 09 07:17:34.337317 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.336926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwkh9\" (UniqueName: \"kubernetes.io/projected/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-kube-api-access-wwkh9\") pod \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\" (UID: \"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9\") "
Sep 09 07:17:34.337317 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.336961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3f7924-4384-41c2-8cfd-a544101159dc-plugins-config-volume\") pod \"5d3f7924-4384-41c2-8cfd-a544101159dc\" (UID: \"5d3f7924-4384-41c2-8cfd-a544101159dc\") "
Sep 09 07:17:34.337643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.337390    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3f7924-4384-41c2-8cfd-a544101159dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d3f7924-4384-41c2-8cfd-a544101159dc" (UID: "5d3f7924-4384-41c2-8cfd-a544101159dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:17:34.337643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.337500    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3f7924-4384-41c2-8cfd-a544101159dc-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:17:34.337643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.337513    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "052ca5a3-9ddc-4a37-a460-6d87edf7b7f9" (UID: "052ca5a3-9ddc-4a37-a460-6d87edf7b7f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:17:34.340302 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.340271    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-kube-api-access-wwkh9" (OuterVolumeSpecName: "kube-api-access-wwkh9") pod "052ca5a3-9ddc-4a37-a460-6d87edf7b7f9" (UID: "052ca5a3-9ddc-4a37-a460-6d87edf7b7f9"). InnerVolumeSpecName "kube-api-access-wwkh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:17:34.340695 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.340671    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3f7924-4384-41c2-8cfd-a544101159dc-kube-api-access-krbdm" (OuterVolumeSpecName: "kube-api-access-krbdm") pod "5d3f7924-4384-41c2-8cfd-a544101159dc" (UID: "5d3f7924-4384-41c2-8cfd-a544101159dc"). InnerVolumeSpecName "kube-api-access-krbdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:17:34.438955 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.438916    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbdm\" (UniqueName: \"kubernetes.io/projected/5d3f7924-4384-41c2-8cfd-a544101159dc-kube-api-access-krbdm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:17:34.438955 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.438952    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:17:34.439197 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:34.438966    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwkh9\" (UniqueName: \"kubernetes.io/projected/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9-kube-api-access-wwkh9\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:17:35.099693 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.099650    2562 generic.go:356] "Generic (PLEG): container finished" podID="052ca5a3-9ddc-4a37-a460-6d87edf7b7f9" containerID="cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd" exitCode=0
Sep 09 07:17:35.100199 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.099703    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"
Sep 09 07:17:35.100199 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.099747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g" event={"ID":"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9","Type":"ContainerDied","Data":"cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd"}
Sep 09 07:17:35.100199 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.099821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g" event={"ID":"052ca5a3-9ddc-4a37-a460-6d87edf7b7f9","Type":"ContainerDied","Data":"f5d6164d75f684054b23f4711b9a422f9e6447380fa4462222b24b14bc40bd33"}
Sep 09 07:17:35.100199 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.099852    2562 scope.go:117] "RemoveContainer" containerID="cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd"
Sep 09 07:17:35.101332 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.101146    2562 generic.go:356] "Generic (PLEG): container finished" podID="5d3f7924-4384-41c2-8cfd-a544101159dc" containerID="5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205" exitCode=0
Sep 09 07:17:35.101332 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.101207    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"
Sep 09 07:17:35.101332 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.101231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d" event={"ID":"5d3f7924-4384-41c2-8cfd-a544101159dc","Type":"ContainerDied","Data":"5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205"}
Sep 09 07:17:35.101332 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.101276    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d" event={"ID":"5d3f7924-4384-41c2-8cfd-a544101159dc","Type":"ContainerDied","Data":"ef76a6129e11ca1b850ebb61e9b61f74ae0316d251496f11cc4c4a524abbd47f"}
Sep 09 07:17:35.110930 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.110855    2562 scope.go:117] "RemoveContainer" containerID="cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd"
Sep 09 07:17:35.111847 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:17:35.111758    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd\": container with ID starting with cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd not found: ID does not exist" containerID="cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd"
Sep 09 07:17:35.111847 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.111795    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd"} err="failed to get container status \"cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd\": rpc error: code = NotFound desc = could not find container \"cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd\": container with ID starting with cd7b4f1ff14e3e8a5b45feaefa9cf8cf0968a22e5913a71555574a88286872dd not found: ID does not exist"
Sep 09 07:17:35.111847 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.111820    2562 scope.go:117] "RemoveContainer" containerID="5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205"
Sep 09 07:17:35.120279 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.120251    2562 scope.go:117] "RemoveContainer" containerID="5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205"
Sep 09 07:17:35.120614 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:17:35.120591    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205\": container with ID starting with 5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205 not found: ID does not exist" containerID="5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205"
Sep 09 07:17:35.120722 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.120620    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205"} err="failed to get container status \"5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205\": rpc error: code = NotFound desc = could not find container \"5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205\": container with ID starting with 5b0198493ca26ec855715fb67ab2fdb09baea98550ff50cd3162b672d1a1a205 not found: ID does not exist"
Sep 09 07:17:35.123401 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.123378    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"]
Sep 09 07:17:35.126996 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.126973    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-36a83e-6c4b6586bb-2gs2g"]
Sep 09 07:17:35.140079 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.140061    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"]
Sep 09 07:17:35.143298 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.143279    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-36a83e-57cc6c67c5-nqp7d"]
Sep 09 07:17:35.230534 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.230502    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052ca5a3-9ddc-4a37-a460-6d87edf7b7f9" path="/var/lib/kubelet/pods/052ca5a3-9ddc-4a37-a460-6d87edf7b7f9/volumes"
Sep 09 07:17:35.230871 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.230858    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d3f7924-4384-41c2-8cfd-a544101159dc" path="/var/lib/kubelet/pods/5d3f7924-4384-41c2-8cfd-a544101159dc/volumes"
Sep 09 07:17:35.231157 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:35.231145    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13d2121-75b7-4982-8507-dd345586a8a6" path="/var/lib/kubelet/pods/e13d2121-75b7-4982-8507-dd345586a8a6/volumes"
Sep 09 07:17:38.369540 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:38.369502    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:17:44.126734 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.126700    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"]
Sep 09 07:17:44.127159 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127071    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13d2121-75b7-4982-8507-dd345586a8a6" containerName="revoke-keys"
Sep 09 07:17:44.127159 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127083    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13d2121-75b7-4982-8507-dd345586a8a6" containerName="revoke-keys"
Sep 09 07:17:44.127159 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127116    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3f7924-4384-41c2-8cfd-a544101159dc" containerName="payload-processing"
Sep 09 07:17:44.127159 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127123    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3f7924-4384-41c2-8cfd-a544101159dc" containerName="payload-processing"
Sep 09 07:17:44.127159 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127144    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052ca5a3-9ddc-4a37-a460-6d87edf7b7f9" containerName="payload-pre-processing"
Sep 09 07:17:44.127159 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127151    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="052ca5a3-9ddc-4a37-a460-6d87edf7b7f9" containerName="payload-pre-processing"
Sep 09 07:17:44.127373 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127207    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="052ca5a3-9ddc-4a37-a460-6d87edf7b7f9" containerName="payload-pre-processing"
Sep 09 07:17:44.127373 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127223    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3f7924-4384-41c2-8cfd-a544101159dc" containerName="payload-processing"
Sep 09 07:17:44.127373 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.127231    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13d2121-75b7-4982-8507-dd345586a8a6" containerName="revoke-keys"
Sep 09 07:17:44.131763 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.131745    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"
Sep 09 07:17:44.134995 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.134970    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:17:44.135321 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.135297    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:17:44.137626 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.137601    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"]
Sep 09 07:17:44.232012 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.231973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065a12b2-c29d-40f5-ba34-59e689f4d901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw\" (UID: \"065a12b2-c29d-40f5-ba34-59e689f4d901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"
Sep 09 07:17:44.333377 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.333340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065a12b2-c29d-40f5-ba34-59e689f4d901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw\" (UID: \"065a12b2-c29d-40f5-ba34-59e689f4d901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"
Sep 09 07:17:44.333953 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.333936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065a12b2-c29d-40f5-ba34-59e689f4d901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw\" (UID: \"065a12b2-c29d-40f5-ba34-59e689f4d901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"
Sep 09 07:17:44.444526 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.444495    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"
Sep 09 07:17:44.714548 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:44.714467    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"]
Sep 09 07:17:44.717634 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:17:44.717606    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065a12b2_c29d_40f5_ba34_59e689f4d901.slice/crio-0dc791948d091be7bcb497b31d8c454e2108eaf380d7df9559e908e62bd5339f WatchSource:0}: Error finding container 0dc791948d091be7bcb497b31d8c454e2108eaf380d7df9559e908e62bd5339f: Status 404 returned error can't find the container with id 0dc791948d091be7bcb497b31d8c454e2108eaf380d7df9559e908e62bd5339f
Sep 09 07:17:45.161578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:45.161486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw" event={"ID":"065a12b2-c29d-40f5-ba34-59e689f4d901","Type":"ContainerStarted","Data":"f8494c8c444af70141f6ff40986d9a9c48ca220961a7092a832d6608e03f4749"}
Sep 09 07:17:45.161578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:45.161540    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw" event={"ID":"065a12b2-c29d-40f5-ba34-59e689f4d901","Type":"ContainerStarted","Data":"0dc791948d091be7bcb497b31d8c454e2108eaf380d7df9559e908e62bd5339f"}
Sep 09 07:17:45.176590 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:45.176545    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw" podStartSLOduration=1.176530499 podStartE2EDuration="1.176530499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:17:45.175164571 +0000 UTC m=+1588.395405684" watchObservedRunningTime="2026-09-09 07:17:45.176530499 +0000 UTC m=+1588.396771669"
Sep 09 07:17:46.168878 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:46.168837    2562 generic.go:356] "Generic (PLEG): container finished" podID="065a12b2-c29d-40f5-ba34-59e689f4d901" containerID="f8494c8c444af70141f6ff40986d9a9c48ca220961a7092a832d6608e03f4749" exitCode=0
Sep 09 07:17:46.169355 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:46.168927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw" event={"ID":"065a12b2-c29d-40f5-ba34-59e689f4d901","Type":"ContainerDied","Data":"f8494c8c444af70141f6ff40986d9a9c48ca220961a7092a832d6608e03f4749"}
Sep 09 07:17:47.306931 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:47.306910    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"
Sep 09 07:17:47.463228 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:47.463141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065a12b2-c29d-40f5-ba34-59e689f4d901-maas-api-service-ca\") pod \"065a12b2-c29d-40f5-ba34-59e689f4d901\" (UID: \"065a12b2-c29d-40f5-ba34-59e689f4d901\") "
Sep 09 07:17:47.463493 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:47.463469    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065a12b2-c29d-40f5-ba34-59e689f4d901-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "065a12b2-c29d-40f5-ba34-59e689f4d901" (UID: "065a12b2-c29d-40f5-ba34-59e689f4d901"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:17:47.564062 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:47.564027    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/065a12b2-c29d-40f5-ba34-59e689f4d901-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:17:48.182181 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:48.182149    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"
Sep 09 07:17:48.182377 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:48.182175    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw" event={"ID":"065a12b2-c29d-40f5-ba34-59e689f4d901","Type":"ContainerDied","Data":"0dc791948d091be7bcb497b31d8c454e2108eaf380d7df9559e908e62bd5339f"}
Sep 09 07:17:48.182377 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:48.182234    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dc791948d091be7bcb497b31d8c454e2108eaf380d7df9559e908e62bd5339f"
Sep 09 07:17:57.168365 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:57.168321    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:17:59.661293 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:59.661253    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"]
Sep 09 07:17:59.661806 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:59.661518    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw" podUID="2b7c285b-7381-4e4f-9965-4cf55e5570e1" containerName="payload-pre-processing" containerID="cri-o://fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be" gracePeriod=30
Sep 09 07:17:59.858642 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:59.858608    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"]
Sep 09 07:17:59.861918 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:59.861894    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bea943-tpxjw"]
Sep 09 07:17:59.915652 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:17:59.915625    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:18:00.084725 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.084692    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7c285b-7381-4e4f-9965-4cf55e5570e1-plugins-config-volume\") pod \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\" (UID: \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\") "
Sep 09 07:18:00.084930 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.084809    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw627\" (UniqueName: \"kubernetes.io/projected/2b7c285b-7381-4e4f-9965-4cf55e5570e1-kube-api-access-dw627\") pod \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\" (UID: \"2b7c285b-7381-4e4f-9965-4cf55e5570e1\") "
Sep 09 07:18:00.085093 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.085066    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7c285b-7381-4e4f-9965-4cf55e5570e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b7c285b-7381-4e4f-9965-4cf55e5570e1" (UID: "2b7c285b-7381-4e4f-9965-4cf55e5570e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:18:00.086926 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.086901    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7c285b-7381-4e4f-9965-4cf55e5570e1-kube-api-access-dw627" (OuterVolumeSpecName: "kube-api-access-dw627") pod "2b7c285b-7381-4e4f-9965-4cf55e5570e1" (UID: "2b7c285b-7381-4e4f-9965-4cf55e5570e1"). InnerVolumeSpecName "kube-api-access-dw627". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:18:00.186019 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.185930    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw627\" (UniqueName: \"kubernetes.io/projected/2b7c285b-7381-4e4f-9965-4cf55e5570e1-kube-api-access-dw627\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:18:00.186019 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.185966    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7c285b-7381-4e4f-9965-4cf55e5570e1-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:18:00.255059 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.255021    2562 generic.go:356] "Generic (PLEG): container finished" podID="2b7c285b-7381-4e4f-9965-4cf55e5570e1" containerID="fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be" exitCode=0
Sep 09 07:18:00.255252 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.255063    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"
Sep 09 07:18:00.255252 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.255131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw" event={"ID":"2b7c285b-7381-4e4f-9965-4cf55e5570e1","Type":"ContainerDied","Data":"fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be"}
Sep 09 07:18:00.255252 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.255185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw" event={"ID":"2b7c285b-7381-4e4f-9965-4cf55e5570e1","Type":"ContainerDied","Data":"2fa3ca818ec3ddf1dfe7f0faba7b582872fe56cef00969e1cb623747931c4214"}
Sep 09 07:18:00.255252 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.255208    2562 scope.go:117] "RemoveContainer" containerID="fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be"
Sep 09 07:18:00.265140 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.265092    2562 scope.go:117] "RemoveContainer" containerID="fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be"
Sep 09 07:18:00.265435 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:18:00.265410    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be\": container with ID starting with fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be not found: ID does not exist" containerID="fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be"
Sep 09 07:18:00.265512 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.265443    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be"} err="failed to get container status \"fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be\": rpc error: code = NotFound desc = could not find container \"fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be\": container with ID starting with fca6ca5707cbdeb4ffc7011d7bd63976084a54ea7c2b866f68b4664d60a132be not found: ID does not exist"
Sep 09 07:18:00.280440 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.280410    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"]
Sep 09 07:18:00.285814 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:00.285793    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bea943-787d99bc74-9wqrw"]
Sep 09 07:18:01.230969 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:01.230943    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065a12b2-c29d-40f5-ba34-59e689f4d901" path="/var/lib/kubelet/pods/065a12b2-c29d-40f5-ba34-59e689f4d901/volumes"
Sep 09 07:18:01.231367 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:01.231296    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b7c285b-7381-4e4f-9965-4cf55e5570e1" path="/var/lib/kubelet/pods/2b7c285b-7381-4e4f-9965-4cf55e5570e1/volumes"
Sep 09 07:18:09.903567 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:09.903529    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:18:58.985424 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:58.985383    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"]
Sep 09 07:18:58.985965 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:58.985601    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs" podUID="1f5dcd88-1336-41fc-b816-e4b2e02416a8" containerName="payload-processing" containerID="cri-o://6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429" gracePeriod=30
Sep 09 07:18:58.991840 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:58.991813    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"]
Sep 09 07:18:58.992081 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:58.992030    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6" podUID="3e7484ed-4695-4cf7-94c4-659378e8bccc" containerName="payload-pre-processing" containerID="cri-o://8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49" gracePeriod=30
Sep 09 07:18:59.239780 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.239716    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:18:59.268710 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.268683    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:18:59.307948 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.307917    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f5dcd88-1336-41fc-b816-e4b2e02416a8-plugins-config-volume\") pod \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\" (UID: \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\") "
Sep 09 07:18:59.307948 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.307956    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6xcw\" (UniqueName: \"kubernetes.io/projected/1f5dcd88-1336-41fc-b816-e4b2e02416a8-kube-api-access-h6xcw\") pod \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\" (UID: \"1f5dcd88-1336-41fc-b816-e4b2e02416a8\") "
Sep 09 07:18:59.308228 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.308083    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7484ed-4695-4cf7-94c4-659378e8bccc-plugins-config-volume\") pod \"3e7484ed-4695-4cf7-94c4-659378e8bccc\" (UID: \"3e7484ed-4695-4cf7-94c4-659378e8bccc\") "
Sep 09 07:18:59.308228 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.308147    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc5h7\" (UniqueName: \"kubernetes.io/projected/3e7484ed-4695-4cf7-94c4-659378e8bccc-kube-api-access-pc5h7\") pod \"3e7484ed-4695-4cf7-94c4-659378e8bccc\" (UID: \"3e7484ed-4695-4cf7-94c4-659378e8bccc\") "
Sep 09 07:18:59.308475 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.308442    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5dcd88-1336-41fc-b816-e4b2e02416a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f5dcd88-1336-41fc-b816-e4b2e02416a8" (UID: "1f5dcd88-1336-41fc-b816-e4b2e02416a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:18:59.308578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.308502    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7484ed-4695-4cf7-94c4-659378e8bccc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e7484ed-4695-4cf7-94c4-659378e8bccc" (UID: "3e7484ed-4695-4cf7-94c4-659378e8bccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:18:59.310057 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.310035    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5dcd88-1336-41fc-b816-e4b2e02416a8-kube-api-access-h6xcw" (OuterVolumeSpecName: "kube-api-access-h6xcw") pod "1f5dcd88-1336-41fc-b816-e4b2e02416a8" (UID: "1f5dcd88-1336-41fc-b816-e4b2e02416a8"). InnerVolumeSpecName "kube-api-access-h6xcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:18:59.310219 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.310198    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7484ed-4695-4cf7-94c4-659378e8bccc-kube-api-access-pc5h7" (OuterVolumeSpecName: "kube-api-access-pc5h7") pod "3e7484ed-4695-4cf7-94c4-659378e8bccc" (UID: "3e7484ed-4695-4cf7-94c4-659378e8bccc"). InnerVolumeSpecName "kube-api-access-pc5h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:18:59.409672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.409639    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f5dcd88-1336-41fc-b816-e4b2e02416a8-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:18:59.409672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.409666    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xcw\" (UniqueName: \"kubernetes.io/projected/1f5dcd88-1336-41fc-b816-e4b2e02416a8-kube-api-access-h6xcw\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:18:59.409672 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.409676    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7484ed-4695-4cf7-94c4-659378e8bccc-plugins-config-volume\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:18:59.409903 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.409684    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc5h7\" (UniqueName: \"kubernetes.io/projected/3e7484ed-4695-4cf7-94c4-659378e8bccc-kube-api-access-pc5h7\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:18:59.604732 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.604628    2562 generic.go:356] "Generic (PLEG): container finished" podID="3e7484ed-4695-4cf7-94c4-659378e8bccc" containerID="8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49" exitCode=0
Sep 09 07:18:59.604732 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.604676    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"
Sep 09 07:18:59.604732 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.604714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6" event={"ID":"3e7484ed-4695-4cf7-94c4-659378e8bccc","Type":"ContainerDied","Data":"8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49"}
Sep 09 07:18:59.604997 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.604771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6" event={"ID":"3e7484ed-4695-4cf7-94c4-659378e8bccc","Type":"ContainerDied","Data":"761f427429ce4c76d88d81d7a29d7e9b8017fd5ab5e8da2815b58179f36ee604"}
Sep 09 07:18:59.604997 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.604793    2562 scope.go:117] "RemoveContainer" containerID="8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49"
Sep 09 07:18:59.606387 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.606224    2562 generic.go:356] "Generic (PLEG): container finished" podID="1f5dcd88-1336-41fc-b816-e4b2e02416a8" containerID="6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429" exitCode=0
Sep 09 07:18:59.606387 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.606272    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"
Sep 09 07:18:59.606387 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.606310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs" event={"ID":"1f5dcd88-1336-41fc-b816-e4b2e02416a8","Type":"ContainerDied","Data":"6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429"}
Sep 09 07:18:59.606387 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.606365    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs" event={"ID":"1f5dcd88-1336-41fc-b816-e4b2e02416a8","Type":"ContainerDied","Data":"b13ec747273da98f024f21a397c9a1c35240dd4fb1432d89177a6f13d688814c"}
Sep 09 07:18:59.614580 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.614559    2562 scope.go:117] "RemoveContainer" containerID="8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49"
Sep 09 07:18:59.614849 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:18:59.614829    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49\": container with ID starting with 8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49 not found: ID does not exist" containerID="8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49"
Sep 09 07:18:59.614943 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.614857    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49"} err="failed to get container status \"8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49\": rpc error: code = NotFound desc = could not find container \"8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49\": container with ID starting with 8c7b7701449367a175906c3a7f89fc8598392c84592cb9f1a2efaf58c457ed49 not found: ID does not exist"
Sep 09 07:18:59.614943 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.614876    2562 scope.go:117] "RemoveContainer" containerID="6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429"
Sep 09 07:18:59.623081 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.623064    2562 scope.go:117] "RemoveContainer" containerID="6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429"
Sep 09 07:18:59.623468 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:18:59.623448    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429\": container with ID starting with 6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429 not found: ID does not exist" containerID="6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429"
Sep 09 07:18:59.623524 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.623475    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429"} err="failed to get container status \"6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429\": rpc error: code = NotFound desc = could not find container \"6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429\": container with ID starting with 6a53c090145a8be341ea0d6c03c319ad3777bafcbfc6e34fb0329dcf0f603429 not found: ID does not exist"
Sep 09 07:18:59.630114 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.630080    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"]
Sep 09 07:18:59.636330 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.636307    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-083e4a-848665779d-bvhf6"]
Sep 09 07:18:59.646763 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.646743    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"]
Sep 09 07:18:59.650916 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:18:59.650892    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-083e4a-dff68877c-s6dzs"]
Sep 09 07:19:01.230005 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:01.229971    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f5dcd88-1336-41fc-b816-e4b2e02416a8" path="/var/lib/kubelet/pods/1f5dcd88-1336-41fc-b816-e4b2e02416a8/volumes"
Sep 09 07:19:01.230462 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:01.230408    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7484ed-4695-4cf7-94c4-659378e8bccc" path="/var/lib/kubelet/pods/3e7484ed-4695-4cf7-94c4-659378e8bccc/volumes"
Sep 09 07:19:02.270817 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:02.270779    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:19:05.619597 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.619565    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"]
Sep 09 07:19:05.620071 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.619918    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy" containerID="cri-o://4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1" gracePeriod=30
Sep 09 07:19:05.965597 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965561    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"]
Sep 09 07:19:05.965904 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965891    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065a12b2-c29d-40f5-ba34-59e689f4d901" containerName="revoke-keys"
Sep 09 07:19:05.965904 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965904    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="065a12b2-c29d-40f5-ba34-59e689f4d901" containerName="revoke-keys"
Sep 09 07:19:05.965997 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965917    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f5dcd88-1336-41fc-b816-e4b2e02416a8" containerName="payload-processing"
Sep 09 07:19:05.965997 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965923    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f5dcd88-1336-41fc-b816-e4b2e02416a8" containerName="payload-processing"
Sep 09 07:19:05.965997 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965937    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7c285b-7381-4e4f-9965-4cf55e5570e1" containerName="payload-pre-processing"
Sep 09 07:19:05.965997 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965943    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7c285b-7381-4e4f-9965-4cf55e5570e1" containerName="payload-pre-processing"
Sep 09 07:19:05.965997 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965950    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7484ed-4695-4cf7-94c4-659378e8bccc" containerName="payload-pre-processing"
Sep 09 07:19:05.965997 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.965956    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7484ed-4695-4cf7-94c4-659378e8bccc" containerName="payload-pre-processing"
Sep 09 07:19:05.966239 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.966011    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7484ed-4695-4cf7-94c4-659378e8bccc" containerName="payload-pre-processing"
Sep 09 07:19:05.966239 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.966019    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f5dcd88-1336-41fc-b816-e4b2e02416a8" containerName="payload-processing"
Sep 09 07:19:05.966239 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.966027    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7c285b-7381-4e4f-9965-4cf55e5570e1" containerName="payload-pre-processing"
Sep 09 07:19:05.966239 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.966034    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="065a12b2-c29d-40f5-ba34-59e689f4d901" containerName="revoke-keys"
Sep 09 07:19:05.970826 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.970807    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"
Sep 09 07:19:05.973450 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.973422    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:19:05.973755 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.973443    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:19:05.976739 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:05.976717    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"]
Sep 09 07:19:06.072304 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:06.072275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45387708-54a7-42d6-85bc-e62ef132e9bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb\" (UID: \"45387708-54a7-42d6-85bc-e62ef132e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"
Sep 09 07:19:06.173538 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:06.173507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45387708-54a7-42d6-85bc-e62ef132e9bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb\" (UID: \"45387708-54a7-42d6-85bc-e62ef132e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"
Sep 09 07:19:06.174137 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:06.174092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45387708-54a7-42d6-85bc-e62ef132e9bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb\" (UID: \"45387708-54a7-42d6-85bc-e62ef132e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"
Sep 09 07:19:06.283042 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:06.282960    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"
Sep 09 07:19:06.562344 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:06.562314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"]
Sep 09 07:19:06.654465 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:06.654433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb" event={"ID":"45387708-54a7-42d6-85bc-e62ef132e9bf","Type":"ContainerStarted","Data":"5e70dc0634f8216a7fc9542f7fa2ec6018aaa128c3286f113028f62c3db72564"}
Sep 09 07:19:07.661874 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:07.661779    2562 generic.go:356] "Generic (PLEG): container finished" podID="45387708-54a7-42d6-85bc-e62ef132e9bf" containerID="4428d103a3d31d52b7fe194f9407412eaa93f0ec2b2e899c22f164a46a69fe32" exitCode=0
Sep 09 07:19:07.662292 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:07.661867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb" event={"ID":"45387708-54a7-42d6-85bc-e62ef132e9bf","Type":"ContainerDied","Data":"4428d103a3d31d52b7fe194f9407412eaa93f0ec2b2e899c22f164a46a69fe32"}
Sep 09 07:19:08.795949 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:08.795923    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"
Sep 09 07:19:08.897193 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:08.897160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45387708-54a7-42d6-85bc-e62ef132e9bf-maas-api-service-ca\") pod \"45387708-54a7-42d6-85bc-e62ef132e9bf\" (UID: \"45387708-54a7-42d6-85bc-e62ef132e9bf\") "
Sep 09 07:19:08.897519 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:08.897495    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45387708-54a7-42d6-85bc-e62ef132e9bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45387708-54a7-42d6-85bc-e62ef132e9bf" (UID: "45387708-54a7-42d6-85bc-e62ef132e9bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:19:08.997899 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:08.997820    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45387708-54a7-42d6-85bc-e62ef132e9bf-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:09.401515 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:09.401427    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 07:19:09.401515 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:09.401491    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 07:19:09.675320 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:09.675294    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"
Sep 09 07:19:09.675499 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:09.675342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb" event={"ID":"45387708-54a7-42d6-85bc-e62ef132e9bf","Type":"ContainerDied","Data":"5e70dc0634f8216a7fc9542f7fa2ec6018aaa128c3286f113028f62c3db72564"}
Sep 09 07:19:09.675499 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:09.675399    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e70dc0634f8216a7fc9542f7fa2ec6018aaa128c3286f113028f62c3db72564"
Sep 09 07:19:10.873581 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:10.873552    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:19:11.014903 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.014866    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-data\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.014903 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.014905    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkj9m\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-kube-api-access-fkj9m\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.015171 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.014928    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-podinfo\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.015171 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.014960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-certs\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.015171 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.014991    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istiod-ca-cert\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.015171 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015068    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-envoy\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.015171 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015147    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-token\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.015171 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015173    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-socket\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.015502 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015205    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-credential-socket\") pod \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\" (UID: \"7e58b0b8-ff48-46d1-b7e5-fa7b63295609\") "
Sep 09 07:19:11.015502 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015225    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:11.015502 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015332    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:11.015502 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015440    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:11.015726 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015537    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:11.015726 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015550    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:19:11.015726 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015598    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.015726 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015616    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.015726 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.015629    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.017331 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.017295    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:11.017331 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.017311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:19:11.017643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.017616    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:19:11.017643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.017619    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-kube-api-access-fkj9m" (OuterVolumeSpecName: "kube-api-access-fkj9m") pod "7e58b0b8-ff48-46d1-b7e5-fa7b63295609" (UID: "7e58b0b8-ff48-46d1-b7e5-fa7b63295609"). InnerVolumeSpecName "kube-api-access-fkj9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:19:11.116216 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.116190    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.116216 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.116213    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.116216 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.116223    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.116440 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.116233    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkj9m\" (UniqueName: \"kubernetes.io/projected/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-kube-api-access-fkj9m\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.116440 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.116241    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.116440 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.116249    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58b0b8-ff48-46d1-b7e5-fa7b63295609-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:11.688506 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.688472    2562 generic.go:356] "Generic (PLEG): container finished" podID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerID="4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1" exitCode=0
Sep 09 07:19:11.688675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.688520    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"
Sep 09 07:19:11.688675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.688549    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" event={"ID":"7e58b0b8-ff48-46d1-b7e5-fa7b63295609","Type":"ContainerDied","Data":"4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1"}
Sep 09 07:19:11.688675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.688602    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8" event={"ID":"7e58b0b8-ff48-46d1-b7e5-fa7b63295609","Type":"ContainerDied","Data":"b347a68e5cd834696de361996b9894254d759a35fbbdd908306e6a025a3fe897"}
Sep 09 07:19:11.688675 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.688626    2562 scope.go:117] "RemoveContainer" containerID="4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1"
Sep 09 07:19:11.698569 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.698545    2562 scope.go:117] "RemoveContainer" containerID="dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a"
Sep 09 07:19:11.709723 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.709699    2562 scope.go:117] "RemoveContainer" containerID="4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1"
Sep 09 07:19:11.709860 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.709839    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"]
Sep 09 07:19:11.709992 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:19:11.709972    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1\": container with ID starting with 4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1 not found: ID does not exist" containerID="4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1"
Sep 09 07:19:11.710035 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.710003    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1"} err="failed to get container status \"4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1\": rpc error: code = NotFound desc = could not find container \"4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1\": container with ID starting with 4d782a8106b68cf933bd3b8cf293f38c12d4b59967bfe195dd1e45a410b745c1 not found: ID does not exist"
Sep 09 07:19:11.710035 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.710022    2562 scope.go:117] "RemoveContainer" containerID="dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a"
Sep 09 07:19:11.710275 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:19:11.710260    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a\": container with ID starting with dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a not found: ID does not exist" containerID="dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a"
Sep 09 07:19:11.710327 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.710280    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a"} err="failed to get container status \"dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a\": rpc error: code = NotFound desc = could not find container \"dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a\": container with ID starting with dd43cf345247b4465322bc6775c71b1a8831c416cd1968997cfbb9b63009de7a not found: ID does not exist"
Sep 09 07:19:11.713680 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:11.713659    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-083e4a-openshift-default-5f6c6d47bb-ntbn8"]
Sep 09 07:19:13.232214 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:13.232178    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" path="/var/lib/kubelet/pods/7e58b0b8-ff48-46d1-b7e5-fa7b63295609/volumes"
Sep 09 07:19:16.476970 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:16.476935    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"]
Sep 09 07:19:16.479834 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:16.479808    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ea0f1c-29815635-mnhv9"]
Sep 09 07:19:16.660410 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:16.660373    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"]
Sep 09 07:19:16.662852 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:16.662824    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ea0f1c-fgmhb"]
Sep 09 07:19:17.230782 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:17.230739    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45387708-54a7-42d6-85bc-e62ef132e9bf" path="/var/lib/kubelet/pods/45387708-54a7-42d6-85bc-e62ef132e9bf/volumes"
Sep 09 07:19:17.231126 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:17.231091    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b81925-92cc-4089-b323-2f45573bb98c" path="/var/lib/kubelet/pods/b8b81925-92cc-4089-b323-2f45573bb98c/volumes"
Sep 09 07:19:19.763326 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:19.763294    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:19:24.049760 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.049722    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"]
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050052    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy"
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050064    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy"
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050084    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy"
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050089    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy"
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050097    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45387708-54a7-42d6-85bc-e62ef132e9bf" containerName="revoke-keys"
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050115    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="45387708-54a7-42d6-85bc-e62ef132e9bf" containerName="revoke-keys"
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050163    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy"
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050169    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e58b0b8-ff48-46d1-b7e5-fa7b63295609" containerName="istio-proxy"
Sep 09 07:19:24.050187 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.050177    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="45387708-54a7-42d6-85bc-e62ef132e9bf" containerName="revoke-keys"
Sep 09 07:19:24.053082 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.053058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"
Sep 09 07:19:24.055652 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.055628    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bbglc\""
Sep 09 07:19:24.056594 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.056578    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 07:19:24.068711 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.068691    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"]
Sep 09 07:19:24.130471 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.130439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94436ad0-6411-43b2-8f41-b2b984ec30d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh\" (UID: \"94436ad0-6411-43b2-8f41-b2b984ec30d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"
Sep 09 07:19:24.232007 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.231964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94436ad0-6411-43b2-8f41-b2b984ec30d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh\" (UID: \"94436ad0-6411-43b2-8f41-b2b984ec30d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"
Sep 09 07:19:24.233624 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.233579    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94436ad0-6411-43b2-8f41-b2b984ec30d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh\" (UID: \"94436ad0-6411-43b2-8f41-b2b984ec30d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"
Sep 09 07:19:24.363212 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.363119    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"
Sep 09 07:19:24.635503 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.635478    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"]
Sep 09 07:19:24.637343 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:19:24.637311    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94436ad0_6411_43b2_8f41_b2b984ec30d6.slice/crio-6c97b7dd0ff9989d2de28621c87548a23c0cb04826c33f1141262683a8f8ac98 WatchSource:0}: Error finding container 6c97b7dd0ff9989d2de28621c87548a23c0cb04826c33f1141262683a8f8ac98: Status 404 returned error can't find the container with id 6c97b7dd0ff9989d2de28621c87548a23c0cb04826c33f1141262683a8f8ac98
Sep 09 07:19:24.767844 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:24.767812    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh" event={"ID":"94436ad0-6411-43b2-8f41-b2b984ec30d6","Type":"ContainerStarted","Data":"6c97b7dd0ff9989d2de28621c87548a23c0cb04826c33f1141262683a8f8ac98"}
Sep 09 07:19:25.775072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:25.775034    2562 generic.go:356] "Generic (PLEG): container finished" podID="94436ad0-6411-43b2-8f41-b2b984ec30d6" containerID="c75c366316b7978220e117c77008a5650900e14632f5ddd464fc97d22081b8ee" exitCode=0
Sep 09 07:19:25.775519 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:25.775141    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh" event={"ID":"94436ad0-6411-43b2-8f41-b2b984ec30d6","Type":"ContainerDied","Data":"c75c366316b7978220e117c77008a5650900e14632f5ddd464fc97d22081b8ee"}
Sep 09 07:19:26.905280 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:26.905256    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"
Sep 09 07:19:27.058722 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:27.058644    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94436ad0-6411-43b2-8f41-b2b984ec30d6-maas-api-service-ca\") pod \"94436ad0-6411-43b2-8f41-b2b984ec30d6\" (UID: \"94436ad0-6411-43b2-8f41-b2b984ec30d6\") "
Sep 09 07:19:27.059021 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:27.059002    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94436ad0-6411-43b2-8f41-b2b984ec30d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94436ad0-6411-43b2-8f41-b2b984ec30d6" (UID: "94436ad0-6411-43b2-8f41-b2b984ec30d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:19:27.160138 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:27.160088    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94436ad0-6411-43b2-8f41-b2b984ec30d6-maas-api-service-ca\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:27.787425 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:27.787393    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"
Sep 09 07:19:27.787592 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:27.787438    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh" event={"ID":"94436ad0-6411-43b2-8f41-b2b984ec30d6","Type":"ContainerDied","Data":"6c97b7dd0ff9989d2de28621c87548a23c0cb04826c33f1141262683a8f8ac98"}
Sep 09 07:19:27.787592 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:27.787490    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c97b7dd0ff9989d2de28621c87548a23c0cb04826c33f1141262683a8f8ac98"
Sep 09 07:19:34.666836 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:34.666790    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"]
Sep 09 07:19:34.669311 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:34.669284    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-988b24-29815635-4ftsm"]
Sep 09 07:19:34.866034 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:34.866003    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"]
Sep 09 07:19:34.868027 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:34.868004    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-988b24-n94gh"]
Sep 09 07:19:35.229907 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:35.229873    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b384d97-f20f-4e40-9ea3-e93fc37b75e9" path="/var/lib/kubelet/pods/8b384d97-f20f-4e40-9ea3-e93fc37b75e9/volumes"
Sep 09 07:19:35.230201 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:35.230187    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94436ad0-6411-43b2-8f41-b2b984ec30d6" path="/var/lib/kubelet/pods/94436ad0-6411-43b2-8f41-b2b984ec30d6/volumes"
Sep 09 07:19:37.263894 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:37.263857    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:19:41.494126 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:41.494024    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"]
Sep 09 07:19:41.494615 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:41.494247    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy" containerID="cri-o://30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1" gracePeriod=30
Sep 09 07:19:44.239374 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:44.239337    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 07:19:44.239761 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:44.239403    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 07:19:46.162859 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.162825    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:19:46.750509 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.750482    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:19:46.820076 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.819963    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-data\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820076 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820018    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn5gg\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-kube-api-access-kn5gg\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820076 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820054    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-podinfo\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820076 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820081    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-envoy\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820479 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820138    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-credential-socket\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820479 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820178    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-socket\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820479 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820206    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8818f8e1-f482-4a72-849c-4a94fb4f7207-istiod-ca-cert\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820479 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820244    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-token\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820479 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820274    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-certs\") pod \"8818f8e1-f482-4a72-849c-4a94fb4f7207\" (UID: \"8818f8e1-f482-4a72-849c-4a94fb4f7207\") "
Sep 09 07:19:46.820479 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820297    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-data" (OuterVolumeSpecName: "istio-data") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:46.820821 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820561    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-data\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.820821 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820653    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8818f8e1-f482-4a72-849c-4a94fb4f7207-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 07:19:46.820821 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820666    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:46.820821 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:46.820821 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.820788    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:46.822357 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.822323    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:19:46.822357 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.822352    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 07:19:46.822795 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.822775    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-kube-api-access-kn5gg" (OuterVolumeSpecName: "kube-api-access-kn5gg") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "kube-api-access-kn5gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:19:46.822870 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.822795    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-token" (OuterVolumeSpecName: "istio-token") pod "8818f8e1-f482-4a72-849c-4a94fb4f7207" (UID: "8818f8e1-f482-4a72-849c-4a94fb4f7207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:19:46.896647 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.896611    2562 generic.go:356] "Generic (PLEG): container finished" podID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerID="30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1" exitCode=0
Sep 09 07:19:46.896770 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.896657    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"
Sep 09 07:19:46.896770 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.896694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" event={"ID":"8818f8e1-f482-4a72-849c-4a94fb4f7207","Type":"ContainerDied","Data":"30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1"}
Sep 09 07:19:46.896770 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.896751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78" event={"ID":"8818f8e1-f482-4a72-849c-4a94fb4f7207","Type":"ContainerDied","Data":"268a0641c99a8f873b96d2f303bccab24f276d09b14d3768d9f64bc7779dedef"}
Sep 09 07:19:46.896890 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.896773    2562 scope.go:117] "RemoveContainer" containerID="30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1"
Sep 09 07:19:46.906763 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.906745    2562 scope.go:117] "RemoveContainer" containerID="d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3"
Sep 09 07:19:46.915435 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.915413    2562 scope.go:117] "RemoveContainer" containerID="30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1"
Sep 09 07:19:46.915716 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:19:46.915695    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1\": container with ID starting with 30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1 not found: ID does not exist" containerID="30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1"
Sep 09 07:19:46.915779 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.915725    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1"} err="failed to get container status \"30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1\": rpc error: code = NotFound desc = could not find container \"30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1\": container with ID starting with 30f338a8f51ccf0a1a0726c9885512dcff73881f7d68a5cec3d78cf5685cc9c1 not found: ID does not exist"
Sep 09 07:19:46.915779 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.915741    2562 scope.go:117] "RemoveContainer" containerID="d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3"
Sep 09 07:19:46.916002 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:19:46.915984    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3\": container with ID starting with d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3 not found: ID does not exist" containerID="d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3"
Sep 09 07:19:46.916044 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.916008    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3"} err="failed to get container status \"d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3\": rpc error: code = NotFound desc = could not find container \"d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3\": container with ID starting with d6929616f03788ab406110f6e361b7c993ec86014a53428255e531f3c2aa6ab3 not found: ID does not exist"
Sep 09 07:19:46.919637 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.919614    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"]
Sep 09 07:19:46.921215 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.921176    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-envoy\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.921215 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.921206    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-credential-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.921366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.921223    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-socket\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.921366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.921240    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8818f8e1-f482-4a72-849c-4a94fb4f7207-istiod-ca-cert\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.921366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.921253    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-token\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.921366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.921266    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8818f8e1-f482-4a72-849c-4a94fb4f7207-workload-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.921366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.921279    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn5gg\" (UniqueName: \"kubernetes.io/projected/8818f8e1-f482-4a72-849c-4a94fb4f7207-kube-api-access-kn5gg\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.921366 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.921292    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8818f8e1-f482-4a72-849c-4a94fb4f7207-istio-podinfo\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:19:46.923181 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:46.923159    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-988b24-openshift-default-677c8fc6cb-8qt78"]
Sep 09 07:19:47.231797 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:19:47.231761    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" path="/var/lib/kubelet/pods/8818f8e1-f482-4a72-849c-4a94fb4f7207/volumes"
Sep 09 07:21:07.570326 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:07.570290    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:21:17.331616 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:17.331580    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:21:17.337365 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:17.337343    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:21:17.649472 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:17.649378    2562 scope.go:117] "RemoveContainer" containerID="c0341c6edc1c685f0eafe78511ddc811fa43037d7c6d32d50ddc09f5f6dc5e2d"
Sep 09 07:21:17.657999 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:17.657977    2562 scope.go:117] "RemoveContainer" containerID="fb07e1786b295ccaf648dbab807e666efb991ed0683c15e4dfa0b0993965f1ba"
Sep 09 07:21:17.665627 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:17.665605    2562 scope.go:117] "RemoveContainer" containerID="f3ccd31876516aeb2f38d6db929f69475e7d1df790d327ff313cd6086f2892d9"
Sep 09 07:21:17.672671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:17.672653    2562 scope.go:117] "RemoveContainer" containerID="b5bbe589fe71a63d739e3ca701f4b229992c74f923ba6a1622bcf5b5536b1865"
Sep 09 07:21:33.461223 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:33.461187    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:21:39.971303 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:39.971263    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:21:45.671012 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:45.670984    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:21:52.969380 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:21:52.969332    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:22:01.762805 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:22:01.762768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:22:08.686151 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:22:08.686115    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:22:16.458348 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:22:16.458309    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:22:17.701476 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:22:17.701444    2562 scope.go:117] "RemoveContainer" containerID="b626c08ac1a1750e24406d23eb64c7ff105c9aaad16e4abc7abeeb20e1d98f0a"
Sep 09 07:22:17.709965 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:22:17.709946    2562 scope.go:117] "RemoveContainer" containerID="0db3f446705fc29e418616054ebacceb9afff68cf2c3e6277899329e5da91117"
Sep 09 07:22:17.717608 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:22:17.717590    2562 scope.go:117] "RemoveContainer" containerID="36ca5ba98e675d09c2722f590a8311654919f69663c48ecf4ef5d97a8091b690"
Sep 09 07:22:24.268417 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:22:24.268379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:23:55.062455 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:23:55.062420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:24:01.769394 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:01.769356    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:24:11.066544 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:11.066455    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:24:16.562856 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.562815    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:24:16.614252 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614224    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"]
Sep 09 07:24:16.614598 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614578    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy"
Sep 09 07:24:16.614598 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614595    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy"
Sep 09 07:24:16.614739 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614607    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy"
Sep 09 07:24:16.614739 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614613    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy"
Sep 09 07:24:16.614739 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614628    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94436ad0-6411-43b2-8f41-b2b984ec30d6" containerName="revoke-keys"
Sep 09 07:24:16.614739 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614633    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="94436ad0-6411-43b2-8f41-b2b984ec30d6" containerName="revoke-keys"
Sep 09 07:24:16.614739 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614699    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="94436ad0-6411-43b2-8f41-b2b984ec30d6" containerName="revoke-keys"
Sep 09 07:24:16.614739 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.614708    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy"
Sep 09 07:24:16.618034 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.618013    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.620378 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.620358    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ebbdbe84-kserve-self-signed-certs\""
Sep 09 07:24:16.630153 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.630130    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"]
Sep 09 07:24:16.741114 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.741077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-dshm\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.741282 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.741130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-home\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.741282 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.741168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-model-cache\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.741282 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.741185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbf4\" (UniqueName: \"kubernetes.io/projected/f5f082b8-b605-4023-86f9-a0ded568a9b7-kube-api-access-wjbf4\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.741282 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.741267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f082b8-b605-4023-86f9-a0ded568a9b7-tls-certs\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.741426 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.741326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-tmp-dir\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.841993 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.841900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-dshm\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.841993 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.841938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-home\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.841993 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.841953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-model-cache\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.841993 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.841974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbf4\" (UniqueName: \"kubernetes.io/projected/f5f082b8-b605-4023-86f9-a0ded568a9b7-kube-api-access-wjbf4\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.842378 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.842051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f082b8-b605-4023-86f9-a0ded568a9b7-tls-certs\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.842378 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.842094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-tmp-dir\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.842497 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.842400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-home\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.842497 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.842457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-model-cache\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.842589 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.842526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-tmp-dir\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.844243 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.844218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-dshm\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.844576 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.844559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f082b8-b605-4023-86f9-a0ded568a9b7-tls-certs\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.850238 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.850215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbf4\" (UniqueName: \"kubernetes.io/projected/f5f082b8-b605-4023-86f9-a0ded568a9b7-kube-api-access-wjbf4\") pod \"e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") " pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:16.926852 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:16.926821    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:17.202859 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:17.202788    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"]
Sep 09 07:24:17.205041 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:24:17.205012    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f082b8_b605_4023_86f9_a0ded568a9b7.slice/crio-57db4e35dc889459c4ce22804a11bfa26d40fee280ef88af648b1b55b0568baa WatchSource:0}: Error finding container 57db4e35dc889459c4ce22804a11bfa26d40fee280ef88af648b1b55b0568baa: Status 404 returned error can't find the container with id 57db4e35dc889459c4ce22804a11bfa26d40fee280ef88af648b1b55b0568baa
Sep 09 07:24:17.356033 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:17.356001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5" event={"ID":"f5f082b8-b605-4023-86f9-a0ded568a9b7","Type":"ContainerStarted","Data":"c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b"}
Sep 09 07:24:17.356033 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:17.356040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5" event={"ID":"f5f082b8-b605-4023-86f9-a0ded568a9b7","Type":"ContainerStarted","Data":"57db4e35dc889459c4ce22804a11bfa26d40fee280ef88af648b1b55b0568baa"}
Sep 09 07:24:17.373959 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:17.373903    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5" podStartSLOduration=1.373886943 podStartE2EDuration="1.373886943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:24:17.372359722 +0000 UTC m=+1980.592600847" watchObservedRunningTime="2026-09-09 07:24:17.373886943 +0000 UTC m=+1980.594128056"
Sep 09 07:24:17.753834 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:17.753802    2562 scope.go:117] "RemoveContainer" containerID="b018302c15c495ef26d4a65c613714ce480312d8e03465db54e2b6fc5d7de2e7"
Sep 09 07:24:17.762056 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:17.762038    2562 scope.go:117] "RemoveContainer" containerID="f8494c8c444af70141f6ff40986d9a9c48ca220961a7092a832d6608e03f4749"
Sep 09 07:24:25.167257 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:25.167217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:24:26.927511 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:26.927473    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:26.928010 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:26.927533    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:26.940042 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:26.940017    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:27.424883 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:27.424858    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:24:45.135918 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:45.135883    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"]
Sep 09 07:24:47.464921 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:47.464880    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:24:55.165183 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:24:55.165148    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:25:00.156234 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.156170    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5" podUID="f5f082b8-b605-4023-86f9-a0ded568a9b7" containerName="main" containerID="cri-o://c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b" gracePeriod=45
Sep 09 07:25:00.408565 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.408498    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:25:00.436488 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-model-cache\") pod \"f5f082b8-b605-4023-86f9-a0ded568a9b7\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") "
Sep 09 07:25:00.436643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-home\") pod \"f5f082b8-b605-4023-86f9-a0ded568a9b7\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") "
Sep 09 07:25:00.436643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436556    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f082b8-b605-4023-86f9-a0ded568a9b7-tls-certs\") pod \"f5f082b8-b605-4023-86f9-a0ded568a9b7\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") "
Sep 09 07:25:00.436643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436606    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjbf4\" (UniqueName: \"kubernetes.io/projected/f5f082b8-b605-4023-86f9-a0ded568a9b7-kube-api-access-wjbf4\") pod \"f5f082b8-b605-4023-86f9-a0ded568a9b7\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") "
Sep 09 07:25:00.436643 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436629    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-dshm\") pod \"f5f082b8-b605-4023-86f9-a0ded568a9b7\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") "
Sep 09 07:25:00.436867 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436676    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-tmp-dir\") pod \"f5f082b8-b605-4023-86f9-a0ded568a9b7\" (UID: \"f5f082b8-b605-4023-86f9-a0ded568a9b7\") "
Sep 09 07:25:00.436867 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436794    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5f082b8-b605-4023-86f9-a0ded568a9b7" (UID: "f5f082b8-b605-4023-86f9-a0ded568a9b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:25:00.436975 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-home" (OuterVolumeSpecName: "home") pod "f5f082b8-b605-4023-86f9-a0ded568a9b7" (UID: "f5f082b8-b605-4023-86f9-a0ded568a9b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:25:00.437037 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.436988    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-model-cache\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:25:00.437037 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.437006    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-home\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:25:00.437037 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.437008    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5f082b8-b605-4023-86f9-a0ded568a9b7" (UID: "f5f082b8-b605-4023-86f9-a0ded568a9b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:25:00.438865 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.438830    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f082b8-b605-4023-86f9-a0ded568a9b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5f082b8-b605-4023-86f9-a0ded568a9b7" (UID: "f5f082b8-b605-4023-86f9-a0ded568a9b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 07:25:00.438865 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.438847    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-dshm" (OuterVolumeSpecName: "dshm") pod "f5f082b8-b605-4023-86f9-a0ded568a9b7" (UID: "f5f082b8-b605-4023-86f9-a0ded568a9b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 07:25:00.439021 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.438917    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f082b8-b605-4023-86f9-a0ded568a9b7-kube-api-access-wjbf4" (OuterVolumeSpecName: "kube-api-access-wjbf4") pod "f5f082b8-b605-4023-86f9-a0ded568a9b7" (UID: "f5f082b8-b605-4023-86f9-a0ded568a9b7"). InnerVolumeSpecName "kube-api-access-wjbf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 07:25:00.538014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.537983    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjbf4\" (UniqueName: \"kubernetes.io/projected/f5f082b8-b605-4023-86f9-a0ded568a9b7-kube-api-access-wjbf4\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:25:00.538014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.538011    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-dshm\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:25:00.538014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.538020    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f082b8-b605-4023-86f9-a0ded568a9b7-tmp-dir\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:25:00.538279 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.538029    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f082b8-b605-4023-86f9-a0ded568a9b7-tls-certs\") on node \"ip-10-0-142-122.ec2.internal\" DevicePath \"\""
Sep 09 07:25:00.589654 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.589615    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5f082b8-b605-4023-86f9-a0ded568a9b7" containerID="c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b" exitCode=0
Sep 09 07:25:00.589820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.589661    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"
Sep 09 07:25:00.589820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.589699    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5" event={"ID":"f5f082b8-b605-4023-86f9-a0ded568a9b7","Type":"ContainerDied","Data":"c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b"}
Sep 09 07:25:00.589820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.589745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5" event={"ID":"f5f082b8-b605-4023-86f9-a0ded568a9b7","Type":"ContainerDied","Data":"57db4e35dc889459c4ce22804a11bfa26d40fee280ef88af648b1b55b0568baa"}
Sep 09 07:25:00.589820 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.589762    2562 scope.go:117] "RemoveContainer" containerID="c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b"
Sep 09 07:25:00.601177 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.601161    2562 scope.go:117] "RemoveContainer" containerID="c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b"
Sep 09 07:25:00.601467 ip-10-0-142-122 kubenswrapper[2562]: E0909 07:25:00.601444    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b\": container with ID starting with c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b not found: ID does not exist" containerID="c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b"
Sep 09 07:25:00.601569 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.601474    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b"} err="failed to get container status \"c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b\": rpc error: code = NotFound desc = could not find container \"c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b\": container with ID starting with c3fcac944fde31fce9d753c6dcdefb2fda5e2ceec0d22af37d102668f7d15f3b not found: ID does not exist"
Sep 09 07:25:00.613634 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.613608    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"]
Sep 09 07:25:00.617048 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:00.617028    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ebbdbe84-kserve-6f49f6b9f-z8bz5"]
Sep 09 07:25:01.235327 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:01.232755    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f082b8-b605-4023-86f9-a0ded568a9b7" path="/var/lib/kubelet/pods/f5f082b8-b605-4023-86f9-a0ded568a9b7/volumes"
Sep 09 07:25:01.666296 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:01.666263    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:25:09.667324 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:09.667287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:25:16.368539 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:16.368503    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:25:17.791763 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:17.791735    2562 scope.go:117] "RemoveContainer" containerID="4428d103a3d31d52b7fe194f9407412eaa93f0ec2b2e899c22f164a46a69fe32"
Sep 09 07:25:22.160186 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:22.160148    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:25:28.763936 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:28.763899    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lv5s8"]
Sep 09 07:25:37.725660 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:37.725628    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-72c9j_b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef/manager/0.log"
Sep 09 07:25:38.085289 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:38.085210    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ggm6p_c4aca7f0-6081-47d3-8195-918da75f5209/manager/0.log"
Sep 09 07:25:38.455401 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:38.455373    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c69cbcb-xbxsv_8ec7696c-49a2-4199-a2e0-e14c1611923a/copy-manifests/0.log"
Sep 09 07:25:38.467511 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:38.467471    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c69cbcb-xbxsv_8ec7696c-49a2-4199-a2e0-e14c1611923a/manager/0.log"
Sep 09 07:25:39.048545 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:39.048501    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815635-sn79w_d979096f-82bd-448b-b195-1a98e1caacbe/cleanup/0.log"
Sep 09 07:25:40.297124 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:40.297037    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-drgw7_ed21057c-9582-46c7-9922-e5e4267ec8f1/kuadrant-console-plugin/0.log"
Sep 09 07:25:40.529875 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:40.529845    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-lv5s8_b83b2514-b189-424c-93ac-2e4b63b0c669/limitador/0.log"
Sep 09 07:25:40.647016 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:40.646928    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-n4b4j_5d007b14-48c5-4efb-a945-dd0f1b73b736/manager/0.log"
Sep 09 07:25:40.994688 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:40.994659    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw_07207665-cc66-4471-819a-c1f6e666e3d5/istio-proxy/0.log"
Sep 09 07:25:41.468998 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:41.468968    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9r9cc_9de492c1-d3e3-4d28-a914-758056b4e6b9/istio-proxy/1.log"
Sep 09 07:25:42.661148 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:42.661121    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbr56g_0a3aa76f-bb1d-4236-918b-bce76c4f97d2/main/0.log"
Sep 09 07:25:42.881718 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:42.881683    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-dd4mx_5af1ef7d-3e54-4354-8fa2-4146edd70968/main/0.log"
Sep 09 07:25:48.993328 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:48.993287    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pv7q6/must-gather-zpcwp"]
Sep 09 07:25:48.993814 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:48.993796    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f082b8-b605-4023-86f9-a0ded568a9b7" containerName="main"
Sep 09 07:25:48.993891 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:48.993816    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f082b8-b605-4023-86f9-a0ded568a9b7" containerName="main"
Sep 09 07:25:48.993954 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:48.993897    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8818f8e1-f482-4a72-849c-4a94fb4f7207" containerName="istio-proxy"
Sep 09 07:25:48.993954 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:48.993912    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f082b8-b605-4023-86f9-a0ded568a9b7" containerName="main"
Sep 09 07:25:48.997202 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:48.997179    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pv7q6/must-gather-zpcwp"
Sep 09 07:25:49.000014 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:48.999993    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pv7q6\"/\"default-dockercfg-zmvt5\""
Sep 09 07:25:49.000147 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:48.999993    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pv7q6\"/\"kube-root-ca.crt\""
Sep 09 07:25:49.001203 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.001175    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pv7q6\"/\"openshift-service-ca.crt\""
Sep 09 07:25:49.007764 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.007741    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pv7q6/must-gather-zpcwp"]
Sep 09 07:25:49.056301 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.056273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmxx\" (UniqueName: \"kubernetes.io/projected/c9be0bcb-d139-4480-9579-da7041d435bd-kube-api-access-lgmxx\") pod \"must-gather-zpcwp\" (UID: \"c9be0bcb-d139-4480-9579-da7041d435bd\") " pod="openshift-must-gather-pv7q6/must-gather-zpcwp"
Sep 09 07:25:49.056426 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.056341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9be0bcb-d139-4480-9579-da7041d435bd-must-gather-output\") pod \"must-gather-zpcwp\" (UID: \"c9be0bcb-d139-4480-9579-da7041d435bd\") " pod="openshift-must-gather-pv7q6/must-gather-zpcwp"
Sep 09 07:25:49.156907 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.156879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9be0bcb-d139-4480-9579-da7041d435bd-must-gather-output\") pod \"must-gather-zpcwp\" (UID: \"c9be0bcb-d139-4480-9579-da7041d435bd\") " pod="openshift-must-gather-pv7q6/must-gather-zpcwp"
Sep 09 07:25:49.157041 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.156950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmxx\" (UniqueName: \"kubernetes.io/projected/c9be0bcb-d139-4480-9579-da7041d435bd-kube-api-access-lgmxx\") pod \"must-gather-zpcwp\" (UID: \"c9be0bcb-d139-4480-9579-da7041d435bd\") " pod="openshift-must-gather-pv7q6/must-gather-zpcwp"
Sep 09 07:25:49.157230 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.157207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9be0bcb-d139-4480-9579-da7041d435bd-must-gather-output\") pod \"must-gather-zpcwp\" (UID: \"c9be0bcb-d139-4480-9579-da7041d435bd\") " pod="openshift-must-gather-pv7q6/must-gather-zpcwp"
Sep 09 07:25:49.165558 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.165532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmxx\" (UniqueName: \"kubernetes.io/projected/c9be0bcb-d139-4480-9579-da7041d435bd-kube-api-access-lgmxx\") pod \"must-gather-zpcwp\" (UID: \"c9be0bcb-d139-4480-9579-da7041d435bd\") " pod="openshift-must-gather-pv7q6/must-gather-zpcwp"
Sep 09 07:25:49.307900 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.307817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pv7q6/must-gather-zpcwp"
Sep 09 07:25:49.575604 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.575579    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pv7q6/must-gather-zpcwp"]
Sep 09 07:25:49.577643 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:25:49.577610    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9be0bcb_d139_4480_9579_da7041d435bd.slice/crio-479e0fdeaa34c7be87ea36401e56224c3aefab7465059a7c49dfce79a0a34359 WatchSource:0}: Error finding container 479e0fdeaa34c7be87ea36401e56224c3aefab7465059a7c49dfce79a0a34359: Status 404 returned error can't find the container with id 479e0fdeaa34c7be87ea36401e56224c3aefab7465059a7c49dfce79a0a34359
Sep 09 07:25:49.579707 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.579691    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 07:25:49.839720 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:49.839633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pv7q6/must-gather-zpcwp" event={"ID":"c9be0bcb-d139-4480-9579-da7041d435bd","Type":"ContainerStarted","Data":"479e0fdeaa34c7be87ea36401e56224c3aefab7465059a7c49dfce79a0a34359"}
Sep 09 07:25:50.851191 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:50.851067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pv7q6/must-gather-zpcwp" event={"ID":"c9be0bcb-d139-4480-9579-da7041d435bd","Type":"ContainerStarted","Data":"68d61ce713ee888901c146695a14f0a54d1d8fbd51152f30aa36bc845e9c2894"}
Sep 09 07:25:50.851191 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:50.851154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pv7q6/must-gather-zpcwp" event={"ID":"c9be0bcb-d139-4480-9579-da7041d435bd","Type":"ContainerStarted","Data":"ad75d2bf52119eda5de03986c947ee309d7646ffc4d2d39082352a8ff941a353"}
Sep 09 07:25:50.868296 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:50.868246    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pv7q6/must-gather-zpcwp" podStartSLOduration=1.9506699140000001 podStartE2EDuration="2.868230755s" totalImagesPullingTime="917.560841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 07:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:25:50.867085064 +0000 UTC m=+2074.087326178" watchObservedRunningTime="2026-09-09 07:25:50.868230755 +0000 UTC m=+2074.088471867"
Sep 09 07:25:52.070317 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:52.070264    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fct4f_2e3268e0-c1e5-45b0-8f4c-81ac05e39d8d/global-pull-secret-syncer/0.log"
Sep 09 07:25:52.199266 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:52.199209    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qzwlw_db7e0b66-9805-4244-884e-c1822ee70b62/konnectivity-agent/0.log"
Sep 09 07:25:52.295122 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:52.295077    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-122.ec2.internal_b67763bc95f0a08ece4881f0237f799d/haproxy/0.log"
Sep 09 07:25:56.667671 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:56.667554    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-drgw7_ed21057c-9582-46c7-9922-e5e4267ec8f1/kuadrant-console-plugin/0.log"
Sep 09 07:25:56.734413 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:56.734361    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-lv5s8_b83b2514-b189-424c-93ac-2e4b63b0c669/limitador/0.log"
Sep 09 07:25:56.804828 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:56.804516    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-n4b4j_5d007b14-48c5-4efb-a945-dd0f1b73b736/manager/0.log"
Sep 09 07:25:58.816249 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:58.816219    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v4zhb_cb1bd352-477f-44c7-99fc-3706c9fd43df/node-exporter/0.log"
Sep 09 07:25:58.837870 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:58.837793    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v4zhb_cb1bd352-477f-44c7-99fc-3706c9fd43df/kube-rbac-proxy/0.log"
Sep 09 07:25:58.863191 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:25:58.863055    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v4zhb_cb1bd352-477f-44c7-99fc-3706c9fd43df/init-textfile/0.log"
Sep 09 07:26:02.636150 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.636089    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"]
Sep 09 07:26:02.641676 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.641640    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.655237 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.655206    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"]
Sep 09 07:26:02.794837 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.794803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-podres\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.795032 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.794845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-lib-modules\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.795032 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.794905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-proc\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.795032 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.794934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vp4\" (UniqueName: \"kubernetes.io/projected/c654da21-3a67-4e83-92df-d29294f11c6b-kube-api-access-t7vp4\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.795032 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.794994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-sys\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.857181 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.857146    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-p6bkh_31c01229-001d-46e7-ae61-1d931257e7b6/volume-data-source-validator/0.log"
Sep 09 07:26:02.896527 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.896430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-proc\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.896798 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.896781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vp4\" (UniqueName: \"kubernetes.io/projected/c654da21-3a67-4e83-92df-d29294f11c6b-kube-api-access-t7vp4\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.897273 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.897249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-sys\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.897448 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.896655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-proc\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.897620 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.897449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-sys\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.897709 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.897541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-podres\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.897837 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.897821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-lib-modules\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.897959 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.897934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-lib-modules\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.898173 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.897648    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c654da21-3a67-4e83-92df-d29294f11c6b-podres\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.906572 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.906541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vp4\" (UniqueName: \"kubernetes.io/projected/c654da21-3a67-4e83-92df-d29294f11c6b-kube-api-access-t7vp4\") pod \"perf-node-gather-daemonset-p8qrd\" (UID: \"c654da21-3a67-4e83-92df-d29294f11c6b\") " pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:02.958884 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:02.958846    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:03.259635 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:03.259593    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"]
Sep 09 07:26:03.268114 ip-10-0-142-122 kubenswrapper[2562]: W0909 07:26:03.268066    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc654da21_3a67_4e83_92df_d29294f11c6b.slice/crio-2701224be895f1db29956ec7d1a8793880236a8015a82bac91a592f098d4c850 WatchSource:0}: Error finding container 2701224be895f1db29956ec7d1a8793880236a8015a82bac91a592f098d4c850: Status 404 returned error can't find the container with id 2701224be895f1db29956ec7d1a8793880236a8015a82bac91a592f098d4c850
Sep 09 07:26:03.954605 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:03.954507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd" event={"ID":"c654da21-3a67-4e83-92df-d29294f11c6b","Type":"ContainerStarted","Data":"4e108c086b6f6f8bcc077b9fb22029b237250d63019de9b6cae71d37adcb1d40"}
Sep 09 07:26:03.954605 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:03.954569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd" event={"ID":"c654da21-3a67-4e83-92df-d29294f11c6b","Type":"ContainerStarted","Data":"2701224be895f1db29956ec7d1a8793880236a8015a82bac91a592f098d4c850"}
Sep 09 07:26:03.955172 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:03.954905    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:03.973504 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:03.973453    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd" podStartSLOduration=1.973428613 podStartE2EDuration="1.973428613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 07:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 07:26:03.970545394 +0000 UTC m=+2087.190786501" watchObservedRunningTime="2026-09-09 07:26:03.973428613 +0000 UTC m=+2087.193669726"
Sep 09 07:26:04.086554 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:04.086519    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rj9m_8f4ffda3-2dfb-47c8-b650-e7a7f7adc082/dns/0.log"
Sep 09 07:26:04.107795 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:04.107757    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rj9m_8f4ffda3-2dfb-47c8-b650-e7a7f7adc082/kube-rbac-proxy/0.log"
Sep 09 07:26:04.221774 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:04.221701    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cp5xk_3cd0276a-00b2-49c9-86a9-4bd63ccf9457/dns-node-resolver/0.log"
Sep 09 07:26:04.818285 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:04.818248    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fjhls_9e27fe24-fa2b-4e20-869b-8c520d98c116/node-ca/0.log"
Sep 09 07:26:05.797680 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:05.797630    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw_07207665-cc66-4471-819a-c1f6e666e3d5/istio-proxy/0.log"
Sep 09 07:26:06.301950 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:06.301907    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9r9cc_9de492c1-d3e3-4d28-a914-758056b4e6b9/istio-proxy/0.log"
Sep 09 07:26:06.312946 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:06.312890    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9r9cc_9de492c1-d3e3-4d28-a914-758056b4e6b9/istio-proxy/1.log"
Sep 09 07:26:07.266400 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:07.266367    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fkqdm_57e258fb-04a0-45c1-82ad-7bde1452a792/serve-healthcheck-canary/0.log"
Sep 09 07:26:07.804578 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:07.804550    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-qc9k2_bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6/insights-operator/0.log"
Sep 09 07:26:07.807220 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:07.804646    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-qc9k2_bb894c77-b7fc-4c06-b4fe-b5c9c9bea8e6/insights-operator/1.log"
Sep 09 07:26:07.919774 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:07.919746    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t555s_5204bfd0-c411-4659-b456-7f7e9ea0c5c6/kube-rbac-proxy/0.log"
Sep 09 07:26:07.946929 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:07.946899    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t555s_5204bfd0-c411-4659-b456-7f7e9ea0c5c6/exporter/0.log"
Sep 09 07:26:07.970296 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:07.970263    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t555s_5204bfd0-c411-4659-b456-7f7e9ea0c5c6/extractor/0.log"
Sep 09 07:26:09.971684 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:09.971657    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pv7q6/perf-node-gather-daemonset-p8qrd"
Sep 09 07:26:10.383498 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:10.383408    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-72c9j_b3ef9da1-c7d6-4e0a-b877-78fdbaef83ef/manager/0.log"
Sep 09 07:26:10.565356 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:10.565318    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ggm6p_c4aca7f0-6081-47d3-8195-918da75f5209/manager/0.log"
Sep 09 07:26:10.753674 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:10.753627    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c69cbcb-xbxsv_8ec7696c-49a2-4199-a2e0-e14c1611923a/manager/0.log"
Sep 09 07:26:10.772944 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:10.772914    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c69cbcb-xbxsv_8ec7696c-49a2-4199-a2e0-e14c1611923a/copy-manifests/0.log"
Sep 09 07:26:12.185600 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:12.185571    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-789c5bdf9c-q4jj2_54d659b4-aea7-49ef-9633-7942680d04fb/manager/0.log"
Sep 09 07:26:12.724208 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:12.724175    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw_07207665-cc66-4471-819a-c1f6e666e3d5/istio-proxy/0.log"
Sep 09 07:26:13.195908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:13.195880    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9r9cc_9de492c1-d3e3-4d28-a914-758056b4e6b9/istio-proxy/0.log"
Sep 09 07:26:13.206603 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:13.206572    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9r9cc_9de492c1-d3e3-4d28-a914-758056b4e6b9/istio-proxy/1.log"
Sep 09 07:26:14.863579 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:14.863503    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-g4fvw_07207665-cc66-4471-819a-c1f6e666e3d5/istio-proxy/0.log"
Sep 09 07:26:15.360819 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:15.360784    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9r9cc_9de492c1-d3e3-4d28-a914-758056b4e6b9/istio-proxy/0.log"
Sep 09 07:26:15.377072 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:15.377038    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9r9cc_9de492c1-d3e3-4d28-a914-758056b4e6b9/istio-proxy/1.log"
Sep 09 07:26:17.388608 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:17.388574    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:26:17.396058 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:17.396037    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnrr4_c8fe6d86-9c3c-45bb-ba16-713c85718256/ovn-acl-logging/0.log"
Sep 09 07:26:17.820650 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:17.820614    2562 scope.go:117] "RemoveContainer" containerID="c75c366316b7978220e117c77008a5650900e14632f5ddd464fc97d22081b8ee"
Sep 09 07:26:18.931524 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:18.931482    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72zrg_24c3a8e5-a25d-4de8-a74d-bb19e329f6f1/kube-multus-additional-cni-plugins/0.log"
Sep 09 07:26:18.953809 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:18.953781    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72zrg_24c3a8e5-a25d-4de8-a74d-bb19e329f6f1/egress-router-binary-copy/0.log"
Sep 09 07:26:18.976281 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:18.976253    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72zrg_24c3a8e5-a25d-4de8-a74d-bb19e329f6f1/cni-plugins/0.log"
Sep 09 07:26:18.995446 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:18.995423    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72zrg_24c3a8e5-a25d-4de8-a74d-bb19e329f6f1/bond-cni-plugin/0.log"
Sep 09 07:26:19.014655 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:19.014632    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72zrg_24c3a8e5-a25d-4de8-a74d-bb19e329f6f1/routeoverride-cni/0.log"
Sep 09 07:26:19.036233 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:19.036210    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72zrg_24c3a8e5-a25d-4de8-a74d-bb19e329f6f1/whereabouts-cni-bincopy/0.log"
Sep 09 07:26:19.059565 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:19.059544    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72zrg_24c3a8e5-a25d-4de8-a74d-bb19e329f6f1/whereabouts-cni/0.log"
Sep 09 07:26:19.303525 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:19.303492    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-f9zfx_40e1d40b-3363-4296-9f1f-873d3d5312aa/kube-multus/0.log"
Sep 09 07:26:19.367908 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:19.367872    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mfdk5_20f229be-0318-40fb-aa2a-62e0df181580/network-metrics-daemon/0.log"
Sep 09 07:26:19.387689 ip-10-0-142-122 kubenswrapper[2562]: I0909 07:26:19.387655    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mfdk5_20f229be-0318-40fb-aa2a-62e0df181580/kube-rbac-proxy/0.log"