Sep 11 03:50:02.008680 ip-10-0-131-122 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 03:50:02.449977 ip-10-0-131-122 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:50:02.449977 ip-10-0-131-122 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 03:50:02.449977 ip-10-0-131-122 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:50:02.449977 ip-10-0-131-122 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 03:50:02.449977 ip-10-0-131-122 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 03:50:02.451131 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.451052    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 03:50:02.455302 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455287    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:02.455302 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455301    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455305    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455309    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455313    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455316    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455319    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455322    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455324    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455327    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455330    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455333    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455336    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455339    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455341    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455344    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455347    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455349    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455352    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455354    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:02.455363 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455357    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455359    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455362    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455365    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455367    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455370    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455374    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455377    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455380    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455382    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455385    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455387    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455390    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455393    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455397    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455401    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455416    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455419    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455422    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:02.455896 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455425    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455428    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455430    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455434    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455436    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455439    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455442    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455444    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455447    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455450    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455453    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455455    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455457    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455461    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455464    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455468    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455472    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455475    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455478    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455481    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:02.456646 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455484    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455486    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455489    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455491    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455494    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455496    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455499    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455501    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455504    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455506    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455509    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455511    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455514    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455517    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455519    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455522    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455525    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455528    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455530    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455533    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:02.457146 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455536    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455539    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455541    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455544    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455548    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455553    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455556    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455559    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455562    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455564    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455566    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455569    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455572    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455574    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455577    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455579    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455582    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455584    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455587    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.455589    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:02.458589 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457223    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457235    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457240    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457243    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457246    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457249    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457252    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457255    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457257    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457260    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457263    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457266    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457269    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457272    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457276    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457280    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457283    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457286    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457289    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457292    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:02.459069 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457295    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457297    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457301    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457304    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457307    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457309    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457312    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457314    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457317    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457319    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457322    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457324    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457327    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457331    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457333    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457336    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457339    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457341    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457344    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457347    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:02.459580 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457349    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457352    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457354    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457357    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457360    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457362    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457365    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457367    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457369    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457372    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457375    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457377    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457381    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457383    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457397    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457400    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457402    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457421    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457426    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457430    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:02.460073 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457433    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457435    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457438    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457441    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457443    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457449    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457454    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457457    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457460    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457463    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457466    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457469    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457472    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457474    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457477    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457480    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457482    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457485    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457487    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:02.460582 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457490    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457492    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457494    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457497    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457501    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457503    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457506    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457509    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457511    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457515    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457518    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457521    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457524    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457527    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457530    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457532    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457535    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457538    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457541    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:02.461082 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457544    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457913    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457917    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457920    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457923    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457925    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457928    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457931    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457934    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457936    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457939    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457941    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457944    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457947    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457949    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457952    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457954    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457957    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457960    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:02.461575 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457963    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457967    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457970    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457974    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457976    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457979    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457982    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457985    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457987    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457990    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457993    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457995    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.457999    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458002    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458004    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458007    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458009    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458012    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458014    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458017    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:02.462031 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458019    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458022    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458024    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458027    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458029    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458032    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458034    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458037    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458039    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458042    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458045    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458048    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458052    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458055    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458058    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458060    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458063    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458066    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458068    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:02.462557 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458070    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458073    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458076    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458078    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458081    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458084    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458087    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458090    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458092    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458095    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458097    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458100    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458102    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458105    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458107    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458110    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458112    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458115    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458117    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458120    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:02.463021 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458124    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458126    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458128    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458131    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458133    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458138    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458140    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458143    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458146    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458148    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458151    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458153    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458155    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458158    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458161    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458163    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458166    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458169    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458172    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458175    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:02.463537 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458177    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458181    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458255    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458295    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458303    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458307    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458311    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458315    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458319    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458323    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458326    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458329    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458332    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458336    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458339    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458342    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458345    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458348    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458352    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458355    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458360    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458363    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458366    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 03:50:02.464071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458369    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458373    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458377    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458380    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458383    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458387    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458390    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458393    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458396    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458399    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458403    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458419    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458422    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458425    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458428    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458431    2566 flags.go:64] FLAG: --enable-server="true"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458434    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458438    2566 flags.go:64] FLAG: --event-burst="100"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458441    2566 flags.go:64] FLAG: --event-qps="50"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458444    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458447    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458450    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458453    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458456    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458459    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 03:50:02.464644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458462    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458465    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458475    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458479    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458483    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458488    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458492    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458497    2566 flags.go:64] FLAG: --feature-gates=""
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458502    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458505    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458508    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458511    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458514    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458518    2566 flags.go:64] FLAG: --help="false"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458521    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458524    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458527    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458530    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458533    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458537    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458540    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458543    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458545    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 03:50:02.465251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458548    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458551    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458554    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458557    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458560    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458562    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458565    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458568    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458571    2566 flags.go:64] FLAG: --lock-file=""
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458573    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458576    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458579    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458585    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458589    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458592    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458595    2566 flags.go:64] FLAG: --logging-format="text"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458598    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458601    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458604    2566 flags.go:64] FLAG: --manifest-url=""
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458607    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458611    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458614    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458619    2566 flags.go:64] FLAG: --max-pods="110"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458622    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458625    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 03:50:02.465853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458628    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458631    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458634    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458637    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458640    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458647    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458650    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458654    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458657    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458660    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458664    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458667    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458670    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458676    2566 flags.go:64] FLAG: --port="10250"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458679    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458682    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06398c848feed7d4f"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458685    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458688    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458691    2566 flags.go:64] FLAG: --register-node="true"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458694    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458697    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458701    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458703    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458706    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458710    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 03:50:02.466479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458713    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458715    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458718    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458721    2566 flags.go:64] FLAG: --runonce="false"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458724    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458728    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458731    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458734    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458737    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458740    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458743    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458746    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458749    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458752    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458755    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458757    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458760    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458763    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458766    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458771    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458775    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458778    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458782    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458785    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458787    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 03:50:02.467110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458790    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458793    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458796    2566 flags.go:64] FLAG: --v="2"
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458800    2566 flags.go:64] FLAG: --version="false"
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458804    2566 flags.go:64] FLAG: --vmodule=""
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458808    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.458811    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458945    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458949    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458953    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458956    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458959    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458964    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458967    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458970    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458973    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458976    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458979    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458981    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458984    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458987    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:02.467787 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458989    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458992    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458995    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.458997    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459000    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459002    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459006    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459009    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459011    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459014    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459016    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459019    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459021    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459024    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459028    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459031    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459035    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459037    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459040    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459042    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:02.468301 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459045    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459048    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459051    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459054    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459057    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459059    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459062    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459066    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459069    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459072    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459075    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459079    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459082    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459084    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459087    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459090    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459093    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459096    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459100    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:02.468814 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459103    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459106    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459108    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459111    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459114    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459116    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459119    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459121    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459124    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459127    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459130    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459133    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459135    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459138    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459140    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459143    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459147    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459150    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459152    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459155    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459157    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:02.469272 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459160    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459162    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459165    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459167    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459170    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459172    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459175    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459177    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459180    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459183    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459187    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459189    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459192    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459194    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459197    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459199    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459202    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459205    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459207    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:02.469811 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459210    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459212    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459214    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459218    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459220    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.459224    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.459229    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.466803    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.466819    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.466835    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.466840    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466895    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466901    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466904    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:02.470269 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466907    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466910    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466913    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466916    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466919    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466922    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466925    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466928    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466931    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466934    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466937    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466940    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466942    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466945    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466948    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466951    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466953    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466956    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466959    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:02.470671 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466961    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466965    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466970    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466973    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466975    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466978    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466980    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466984    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466987    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466990    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466993    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466995    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.466998    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467001    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467003    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467006    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467009    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467012    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467014    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467017    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:02.471186 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467020    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467022    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467025    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467028    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467031    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467034    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467037    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467040    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467042    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467045    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467048    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467051    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467053    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467055    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467058    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467061    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467064    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467067    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467070    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467072    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:02.471696 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467075    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467078    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467081    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467084    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467086    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467089    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467091    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467094    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467097    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467101    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467104    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467108    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467111    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467114    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467117    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467121    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467125    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467128    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467131    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:02.472191 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467134    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467136    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467139    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467141    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467144    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467147    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467149    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467152    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467154    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467157    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467161    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467164    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467168    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467171    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467174    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467177    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467180    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:02.472669 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467182    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.467187    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467295    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467300    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467303    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467306    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467309    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467312    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467315    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467318    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467320    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467323    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467326    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467329    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467331    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 03:50:02.473088 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467334    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467337    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467339    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467342    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467344    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467347    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467349    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467352    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467356    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467360    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467363    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467366    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467368    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467371    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467375    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467378    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467381    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467384    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467387    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 03:50:02.473482 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467390    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467393    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467396    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467398    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467402    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467417    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467419    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467422    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467424    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467427    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467429    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467432    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467434    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467437    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467439    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467442    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467445    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467447    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467450    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467452    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 03:50:02.473953 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467455    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467457    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467460    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467462    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467465    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467467    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467470    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467472    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467475    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467478    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467481    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467485    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467488    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467491    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467494    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467497    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467500    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467502    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467505    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 03:50:02.474498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467507    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467510    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467512    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467515    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467518    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467520    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467522    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467525    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467528    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467530    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467533    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467535    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467538    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467540    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467543    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467545    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467548    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467551    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467553    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 03:50:02.474958 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467556    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467558    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467560    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467563    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467566    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467569    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467571    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467574    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:02.467577    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.467582    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.467679    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.470926    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.471987    2566 server.go:1019] "Starting client certificate rotation"
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.472085    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:50:02.475424 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.472123    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 03:50:02.475773 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.474850    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:50:02.479741 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.479726    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 03:50:02.488015 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.487994    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 11 03:50:02.491202 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.491185    2566 log.go:25] "Validated CRI v1 image API"
Sep 11 03:50:02.492975 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.492954    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 03:50:02.498886 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.498866    2566 fs.go:135] Filesystem UUIDs: map[48c8141e-a8b2-4afb-ad36-2d708176f7e8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ff47a303-6243-48ab-afa7-4ad278233165:/dev/nvme0n1p3]
Sep 11 03:50:02.498959 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.498885    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 03:50:02.501303 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.501288    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:50:02.504919 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.504816    2566 manager.go:216] Machine: {Timestamp:2026-09-11 03:50:02.503418943 +0000 UTC m=+0.369438669 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102824 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24cc94d326c5f4d99a613366b29cf4 SystemUUID:ec24cc94-d326-c5f4-d99a-613366b29cf4 BootID:dcca5763-d566-49ec-b89d-954738901554 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d5:05:88:a0:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d5:05:88:a0:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:55:f5:f2:c9:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 03:50:02.504919 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.504913    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 03:50:02.505026 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.504982    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 03:50:02.508402 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.508381    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 03:50:02.508535 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.508414    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 03:50:02.509307 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.509298    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 03:50:02.509375 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.509309    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 03:50:02.509375 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.509317    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:50:02.510121 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.510111    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 03:50:02.511469 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.511460    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 03:50:02.515014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515003    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:50:02.515125 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515116    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 03:50:02.515158 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515144    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 03:50:02.515158 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515153    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 03:50:02.515215 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515167    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 03:50:02.515215 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515176    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 11 03:50:02.515215 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515185    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 03:50:02.515498 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515485    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:50:02.515556 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.515507    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 03:50:02.517707 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.517690    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 03:50:02.521731 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.521716    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gn7tj"
Sep 11 03:50:02.523379 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.523367    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 03:50:02.523437 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.523396    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 03:50:02.525556 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525544    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525560    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525566    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525571    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525584    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525590    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525603    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525608    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525615    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 03:50:02.525623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525620    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 03:50:02.525854 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525634    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 03:50:02.525854 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.525640    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 03:50:02.526428 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.526382    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 03:50:02.526547 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.526531    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 03:50:02.526591 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.526579    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 03:50:02.526591 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.526588    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 03:50:02.529064 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.529048    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gn7tj"
Sep 11 03:50:02.531330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.531312    2566 server.go:1268] "Started kubelet"
Sep 11 03:50:02.532139 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.531380    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 03:50:02.532139 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.532095    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 03:50:02.532901 ip-10-0-131-122 systemd[1]: Started Kubernetes Kubelet.
Sep 11 03:50:02.533039 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.532898    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 03:50:02.534916 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.534881    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 03:50:02.536747 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.536734    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 03:50:02.537480 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.537245    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 03:50:02.540241 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.540189    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 03:50:02.544320 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.544304    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 03:50:02.544320 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.544314    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 03:50:02.544467 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.544365    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:02.545019 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545002    2566 factory.go:55] Registering systemd factory
Sep 11 03:50:02.545121 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545043    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 11 03:50:02.545121 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545096    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 03:50:02.545121 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545110    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 03:50:02.545265 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545220    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 03:50:02.545265 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545225    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 03:50:02.545265 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545243    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 03:50:02.545422 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545323    2566 factory.go:153] Registering CRI-O factory
Sep 11 03:50:02.545422 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545348    2566 factory.go:223] Registration of the crio container factory successfully
Sep 11 03:50:02.545492 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545420    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 03:50:02.545492 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545447    2566 factory.go:103] Registering Raw factory
Sep 11 03:50:02.545492 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545463    2566 manager.go:1196] Started watching for new ooms in manager
Sep 11 03:50:02.545820 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.545808    2566 manager.go:349] Starting recovery of all containers
Sep 11 03:50:02.550461 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.549461    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:02.552903 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.552737    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-122.ec2.internal\" not found" node="ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.557518 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.557462    2566 manager.go:354] Recovery completed
Sep 11 03:50:02.563011 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.562996    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:02.565110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.565079    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:02.565110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.565109    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:02.565227 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.565123    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:02.566173 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.566151    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 03:50:02.566252 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.566178    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 03:50:02.566252 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.566202    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 03:50:02.566435 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.566424    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:02.567636 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.567615    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:02.567732 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.567719    2566 policy_none.go:49] "None policy: Start"
Sep 11 03:50:02.567780 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.567752    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:02.567780 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.567754    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 03:50:02.567854 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.567789    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 03:50:02.567854 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.567794    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:02.569079 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.569067    2566 policy_none.go:47] "Start"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.603092    2566 manager.go:337] "Starting Device Plugin manager"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.603127    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.603140    2566 server.go:87] "Starting device plugin registration server"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.603451    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.603461    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.603587    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.603658    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.603667    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.604194    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.604249    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.604489    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.605639    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.605777    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.605801    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.605843    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 03:50:02.613853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.607560    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:02.704247 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.704182    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:02.706536 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.706518    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal"]
Sep 11 03:50:02.706592 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.706586    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:02.708083 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.708070    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:02.708150 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.708072    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:02.708150 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.708100    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:02.708150 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.708115    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:02.708150 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.708128    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:02.708150 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.708148    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.708150 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.708116    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:02.710359 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.710347    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:02.710510 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.710498    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.710549 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.710524    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:02.711024 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.711010    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:02.711122 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.711030    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:02.711122 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.711039    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:02.711122 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.711052    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:02.711122 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.711074    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:02.711122 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.711116    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:02.713214 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.713200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.713271 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.713225    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 03:50:02.713896 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.713881    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 03:50:02.713980 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.713908    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 03:50:02.713980 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.713924    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeHasSufficientPID"
Sep 11 03:50:02.716674 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.716661    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.716721 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.716679    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-122.ec2.internal\": node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:02.730042 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.730027    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:02.746891 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.746875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e7e89872af809b35925e7ce91f29761-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal\" (UID: \"9e7e89872af809b35925e7ce91f29761\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.746978 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.746898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7e89872af809b35925e7ce91f29761-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal\" (UID: \"9e7e89872af809b35925e7ce91f29761\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.746978 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.746908    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-122.ec2.internal\" not found" node="ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.746978 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.746969    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/147e739407ba5f1c7ea961026966fc31-config\") pod \"kube-apiserver-proxy-ip-10-0-131-122.ec2.internal\" (UID: \"147e739407ba5f1c7ea961026966fc31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.751064 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.751046    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-122.ec2.internal\" not found" node="ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.830965 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.830946    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:02.847857 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.847840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e7e89872af809b35925e7ce91f29761-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal\" (UID: \"9e7e89872af809b35925e7ce91f29761\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.847917 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.847861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7e89872af809b35925e7ce91f29761-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal\" (UID: \"9e7e89872af809b35925e7ce91f29761\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.847917 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.847879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/147e739407ba5f1c7ea961026966fc31-config\") pod \"kube-apiserver-proxy-ip-10-0-131-122.ec2.internal\" (UID: \"147e739407ba5f1c7ea961026966fc31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.847992 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.847923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/147e739407ba5f1c7ea961026966fc31-config\") pod \"kube-apiserver-proxy-ip-10-0-131-122.ec2.internal\" (UID: \"147e739407ba5f1c7ea961026966fc31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.847992 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.847940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e7e89872af809b35925e7ce91f29761-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal\" (UID: \"9e7e89872af809b35925e7ce91f29761\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.847992 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:02.847949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7e89872af809b35925e7ce91f29761-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal\" (UID: \"9e7e89872af809b35925e7ce91f29761\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:02.931464 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:02.931433    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:03.032202 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.032133    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:03.050526 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.050509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:03.056704 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.055821    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:03.132243 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.132203    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:03.232683 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.232652    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:03.333215 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.333152    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:03.433649 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.433621    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-122.ec2.internal\" not found"
Sep 11 03:50:03.436631 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.436613    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:03.445496 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.445471    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:03.456595 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.456577    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:50:03.458443 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.458429    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal"
Sep 11 03:50:03.467415 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.467399    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 03:50:03.471678 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.471662    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 03:50:03.471790 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.471773    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:50:03.471850 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.471809    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:50:03.471850 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.471829    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 03:50:03.516279 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.516259    2566 apiserver.go:52] "Watching apiserver"
Sep 11 03:50:03.535549 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.534846    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 03:50:03.536387 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.536341    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 03:45:03 +0000 UTC" deadline="2028-06-23 15:54:14.424222848 +0000 UTC"
Sep 11 03:50:03.536387 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.536386    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15636h4m10.88784185s"
Sep 11 03:50:03.537937 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.537901    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-qqc4r","openshift-dns/node-resolver-2d7rk","openshift-image-registry/node-ca-6lvvf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal","openshift-multus/multus-additional-cni-plugins-h2mf6","kube-system/konnectivity-agent-xh7zf","openshift-multus/multus-c4lfh","openshift-multus/network-metrics-daemon-k57hw","openshift-network-diagnostics/network-check-target-t7pq9","openshift-network-operator/iptables-alerter-crz62","openshift-ovn-kubernetes/ovnkube-node-vkpvh","kube-system/global-pull-secret-syncer-frxxs","kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"]
Sep 11 03:50:03.540859 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.540840    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.543880 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.543858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.544528 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.544469    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 03:50:03.545092 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.545077    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:50:03.545274 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.545262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fdgwq\""
Sep 11 03:50:03.545321 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.545311    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 03:50:03.545584 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.545574    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 03:50:03.546236 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.546219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 03:50:03.546702 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.546681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.547792 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.547605    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l77nx\""
Sep 11 03:50:03.547792 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.547643    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 03:50:03.547792 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.547670    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 03:50:03.547792 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.547612    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 03:50:03.547792 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.547643    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 03:50:03.547792 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.547674    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 03:50:03.548785 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.548770    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 03:50:03.549306 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.549291    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 03:50:03.549306 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.549299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ghklm\""
Sep 11 03:50:03.549450 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.549295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.549705 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.549690    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 03:50:03.551327 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.551302    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 03:50:03.551431 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.551399    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 03:50:03.551667 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.551644    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 03:50:03.551742 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.551716    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.551808 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.551762    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p9xjd\""
Sep 11 03:50:03.552792 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.552770    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-var-lib-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.552960 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.552809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.552960 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.552834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovnkube-script-lib\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.552960 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.552889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-kubelet\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.552960 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.552945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553166 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.552972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65xq\" (UniqueName: \"kubernetes.io/projected/f46f4324-94b3-488f-bc24-d5af8a3740ae-kube-api-access-m65xq\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.553166 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-cni-netd\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553166 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc800d9-e949-4c5e-9804-1f74c6b507b9-host\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.553166 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-ovn\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553166 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-cni-bin\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553166 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovnkube-config\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-env-overrides\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rx2z\" (UniqueName: \"kubernetes.io/projected/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-kube-api-access-5rx2z\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f4324-94b3-488f-bc24-d5af8a3740ae-host-slash\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-slash\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovn-node-metrics-cert\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f46f4324-94b3-488f-bc24-d5af8a3740ae-iptables-alerter-script\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-systemd\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-etc-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-systemd-units\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-node-log\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.553491 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-log-socket\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.554153 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtth\" (UniqueName: \"kubernetes.io/projected/cdc800d9-e949-4c5e-9804-1f74c6b507b9-kube-api-access-lrtth\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.554153 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-run-netns\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.554153 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdc800d9-e949-4c5e-9804-1f74c6b507b9-serviceca\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.554153 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v4qq8\""
Sep 11 03:50:03.554153 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 03:50:03.554153 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.553922    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 03:50:03.554537 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.554197    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 03:50:03.554537 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.554361    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 03:50:03.554537 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.554369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:03.554537 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.554373    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 03:50:03.554803 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.554779    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 03:50:03.556450 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.556432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:03.556532 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.556438    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 03:50:03.556532 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.556498    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:03.556637 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.556597    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 03:50:03.556637 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.556604    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gqkq8\""
Sep 11 03:50:03.559573 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.559557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:03.559647 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.559615    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:03.562208 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.562185    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.564928 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.564907    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 03:50:03.564928 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.564917    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 03:50:03.565077 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.564959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-75kvp\""
Sep 11 03:50:03.565622 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.565600    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.569600 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.569578    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 03:50:03.569926 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.569901    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:50:03.571866 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.571845    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.572306 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.572289    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gc9rn\""
Sep 11 03:50:03.574143 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.574126    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lxgqb\""
Sep 11 03:50:03.574868 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.574850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.574954 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.574913    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:03.576386 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.576369    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 03:50:03.580300 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.580283    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lpg2g"
Sep 11 03:50:03.588238 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.588223    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lpg2g"
Sep 11 03:50:03.633499 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.633472    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147e739407ba5f1c7ea961026966fc31.slice/crio-e1f52a2602738dba0ee2fec8850e76f0c4a0bd5e7967c0eee5096bffad6fee83 WatchSource:0}: Error finding container e1f52a2602738dba0ee2fec8850e76f0c4a0bd5e7967c0eee5096bffad6fee83: Status 404 returned error can't find the container with id e1f52a2602738dba0ee2fec8850e76f0c4a0bd5e7967c0eee5096bffad6fee83
Sep 11 03:50:03.633680 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.633665    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7e89872af809b35925e7ce91f29761.slice/crio-c4a829fe715213ae66a9bbd25e6316909b7d309fa534738bec44175d01f6e689 WatchSource:0}: Error finding container c4a829fe715213ae66a9bbd25e6316909b7d309fa534738bec44175d01f6e689: Status 404 returned error can't find the container with id c4a829fe715213ae66a9bbd25e6316909b7d309fa534738bec44175d01f6e689
Sep 11 03:50:03.640016 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.640001    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:50:03.646638 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.646619    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 03:50:03.654360 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-run-netns\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.654462 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdc800d9-e949-4c5e-9804-1f74c6b507b9-serviceca\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.654462 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-registration-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.654606 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvzj\" (UniqueName: \"kubernetes.io/projected/b98300df-7e61-45ad-87d0-b0185bb00771-kube-api-access-ncvzj\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.654606 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-run-netns\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.654606 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-var-lib-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.654765 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovnkube-script-lib\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.654765 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-var-lib-kubelet\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.654765 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b6be1d-bba4-48f0-900a-a36c80597729-cni-binary-copy\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.654765 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27pd\" (UniqueName: \"kubernetes.io/projected/82a364ce-5981-412b-9cf0-65da78af29d9-kube-api-access-j27pd\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.654765 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-kubelet\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.654765 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-var-lib-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-host\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-cnibin\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65xq\" (UniqueName: \"kubernetes.io/projected/f46f4324-94b3-488f-bc24-d5af8a3740ae-kube-api-access-m65xq\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc800d9-e949-4c5e-9804-1f74c6b507b9-host\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-socket-dir-parent\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfl2j\" (UniqueName: \"kubernetes.io/projected/e1b6be1d-bba4-48f0-900a-a36c80597729-kube-api-access-bfl2j\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.654998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdc800d9-e949-4c5e-9804-1f74c6b507b9-serviceca\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-socket-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.655108 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655139    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-kubelet\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc800d9-e949-4c5e-9804-1f74c6b507b9-host\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-cni-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-sys-fs\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovnkube-script-lib\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cd262c2-7fea-463a-9c9d-84134976390d-agent-certs\") pod \"konnectivity-agent-xh7zf\" (UID: \"3cd262c2-7fea-463a-9c9d-84134976390d\") " pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a364ce-5981-412b-9cf0-65da78af29d9-tmp-dir\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f4324-94b3-488f-bc24-d5af8a3740ae-host-slash\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-slash\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovn-node-metrics-cert\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f4324-94b3-488f-bc24-d5af8a3740ae-host-slash\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysctl-d\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-netns\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-slash\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-cni-bin\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f46f4324-94b3-488f-bc24-d5af8a3740ae-iptables-alerter-script\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.655693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-systemd\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysconfig\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-systemd\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655570    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97pb\" (UniqueName: \"kubernetes.io/projected/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-kube-api-access-m97pb\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-systemd-units\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-node-log\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-systemd-units\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37437296-da5a-4a64-8dca-a89fa0263d79-dbus\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.655998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-node-log\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656021    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-systemd\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-daemon-config\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.656675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-device-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f46f4324-94b3-488f-bc24-d5af8a3740ae-iptables-alerter-script\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-cnibin\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68svw\" (UniqueName: \"kubernetes.io/projected/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-kube-api-access-68svw\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-modprobe-d\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-cni-multus\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-hostroot\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cd262c2-7fea-463a-9c9d-84134976390d-konnectivity-ca\") pod \"konnectivity-agent-xh7zf\" (UID: \"3cd262c2-7fea-463a-9c9d-84134976390d\") " pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-cni-netd\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-lib-modules\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-system-cni-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-cni-netd\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-kubelet\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.657414 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-ovn\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-cni-bin\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovnkube-config\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-env-overrides\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-run-ovn\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-kubernetes\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-sys\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-host-cni-bin\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-tmp\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-conf-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rx2z\" (UniqueName: \"kubernetes.io/projected/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-kube-api-access-5rx2z\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37437296-da5a-4a64-8dca-a89fa0263d79-kubelet-config\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwnbx\" (UniqueName: \"kubernetes.io/projected/28be8a53-5cb9-4843-bcaa-5b840e135f08-kube-api-access-qwnbx\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82a364ce-5981-412b-9cf0-65da78af29d9-hosts-file\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-system-cni-dir\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-etc-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.657955 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-run\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-env-overrides\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-tuned\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-etc-openvswitch\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.656988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-k8s-cni-cncf-io\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-multus-certs\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-etc-kubernetes\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovnkube-config\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-log-socket\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657136    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtth\" (UniqueName: \"kubernetes.io/projected/cdc800d9-e949-4c5e-9804-1f74c6b507b9-kube-api-access-lrtth\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysctl-conf\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.658507 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.657198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-log-socket\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.660095 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.659978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-ovn-node-metrics-cert\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.663834 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.663813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65xq\" (UniqueName: \"kubernetes.io/projected/f46f4324-94b3-488f-bc24-d5af8a3740ae-kube-api-access-m65xq\") pod \"iptables-alerter-crz62\" (UID: \"f46f4324-94b3-488f-bc24-d5af8a3740ae\") " pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.664915 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.664896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rx2z\" (UniqueName: \"kubernetes.io/projected/d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a-kube-api-access-5rx2z\") pod \"ovnkube-node-vkpvh\" (UID: \"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.665536 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.665468    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtth\" (UniqueName: \"kubernetes.io/projected/cdc800d9-e949-4c5e-9804-1f74c6b507b9-kube-api-access-lrtth\") pod \"node-ca-6lvvf\" (UID: \"cdc800d9-e949-4c5e-9804-1f74c6b507b9\") " pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.757811 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-cnibin\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.757945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68svw\" (UniqueName: \"kubernetes.io/projected/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-kube-api-access-68svw\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:03.757945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-modprobe-d\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.757945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:03.757945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-cni-multus\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.757945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-hostroot\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.757945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cd262c2-7fea-463a-9c9d-84134976390d-konnectivity-ca\") pod \"konnectivity-agent-xh7zf\" (UID: \"3cd262c2-7fea-463a-9c9d-84134976390d\") " pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:03.757945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-lib-modules\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.757945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-cnibin\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-system-cni-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-hostroot\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-cni-multus\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.757991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-kubelet\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.758025    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-lib-modules\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-kubernetes\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-system-cni-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-kubelet\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-kubernetes\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-modprobe-d\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.758107    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs podName:c6f66daa-4ff1-45b4-93d6-cc8de58802b3 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:04.258065444 +0000 UTC m=+2.124085176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs") pod "network-metrics-daemon-k57hw" (UID: "c6f66daa-4ff1-45b4-93d6-cc8de58802b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-sys\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-tmp\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.758346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-conf-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37437296-da5a-4a64-8dca-a89fa0263d79-kubelet-config\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-sys\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwnbx\" (UniqueName: \"kubernetes.io/projected/28be8a53-5cb9-4843-bcaa-5b840e135f08-kube-api-access-qwnbx\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-conf-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82a364ce-5981-412b-9cf0-65da78af29d9-hosts-file\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37437296-da5a-4a64-8dca-a89fa0263d79-kubelet-config\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758378    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82a364ce-5981-412b-9cf0-65da78af29d9-hosts-file\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-system-cni-dir\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-run\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-tuned\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758532    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3cd262c2-7fea-463a-9c9d-84134976390d-konnectivity-ca\") pod \"konnectivity-agent-xh7zf\" (UID: \"3cd262c2-7fea-463a-9c9d-84134976390d\") " pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-system-cni-dir\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758537    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-k8s-cni-cncf-io\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-k8s-cni-cncf-io\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-multus-certs\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-multus-certs\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-run\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.759245 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-etc-kubernetes\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-etc-kubernetes\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysctl-conf\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-registration-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvzj\" (UniqueName: \"kubernetes.io/projected/b98300df-7e61-45ad-87d0-b0185bb00771-kube-api-access-ncvzj\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-var-lib-kubelet\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b6be1d-bba4-48f0-900a-a36c80597729-cni-binary-copy\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysctl-conf\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27pd\" (UniqueName: \"kubernetes.io/projected/82a364ce-5981-412b-9cf0-65da78af29d9-kube-api-access-j27pd\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-registration-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-host\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-var-lib-kubelet\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-cnibin\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.760144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758958    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-host\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-socket-dir-parent\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.758998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfl2j\" (UniqueName: \"kubernetes.io/projected/e1b6be1d-bba4-48f0-900a-a36c80597729-kube-api-access-bfl2j\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-socket-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-cni-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-socket-dir-parent\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-sys-fs\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-sys-fs\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-cni-dir\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-socket-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cd262c2-7fea-463a-9c9d-84134976390d-agent-certs\") pod \"konnectivity-agent-xh7zf\" (UID: \"3cd262c2-7fea-463a-9c9d-84134976390d\") " pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a364ce-5981-412b-9cf0-65da78af29d9-tmp-dir\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysctl-d\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-netns\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-cni-bin\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.760791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysconfig\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b6be1d-bba4-48f0-900a-a36c80597729-cni-binary-copy\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97pb\" (UniqueName: \"kubernetes.io/projected/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-kube-api-access-m97pb\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-run-netns\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37437296-da5a-4a64-8dca-a89fa0263d79-dbus\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysctl-d\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-systemd\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759715    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a364ce-5981-412b-9cf0-65da78af29d9-tmp-dir\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-systemd\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-daemon-config\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-cnibin\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-device-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b6be1d-bba4-48f0-900a-a36c80597729-host-var-lib-cni-bin\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.759780    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:03.761319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b98300df-7e61-45ad-87d0-b0185bb00771-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-sysconfig\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37437296-da5a-4a64-8dca-a89fa0263d79-dbus\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.759852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28be8a53-5cb9-4843-bcaa-5b840e135f08-device-dir\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.759894    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret podName:37437296-da5a-4a64-8dca-a89fa0263d79 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:04.259876477 +0000 UTC m=+2.125896209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret") pod "global-pull-secret-syncer-frxxs" (UID: "37437296-da5a-4a64-8dca-a89fa0263d79") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.760052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b98300df-7e61-45ad-87d0-b0185bb00771-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.760131    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1b6be1d-bba4-48f0-900a-a36c80597729-multus-daemon-config\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.760968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-tmp\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.761803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3cd262c2-7fea-463a-9c9d-84134976390d-agent-certs\") pod \"konnectivity-agent-xh7zf\" (UID: \"3cd262c2-7fea-463a-9c9d-84134976390d\") " pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:03.762040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.761824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-etc-tuned\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.772897 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.772877    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:03.772988 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.772902    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:03.772988 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.772942    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ls7w4 for pod openshift-network-diagnostics/network-check-target-t7pq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:03.773088 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:03.773001    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4 podName:544a104a-4499-4f6e-a751-77db1ae1aca2 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:04.272986069 +0000 UTC m=+2.139005811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ls7w4" (UniqueName: "kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4") pod "network-check-target-t7pq9" (UID: "544a104a-4499-4f6e-a751-77db1ae1aca2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:03.774346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.774319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwnbx\" (UniqueName: \"kubernetes.io/projected/28be8a53-5cb9-4843-bcaa-5b840e135f08-kube-api-access-qwnbx\") pod \"aws-ebs-csi-driver-node-6zm7x\" (UID: \"28be8a53-5cb9-4843-bcaa-5b840e135f08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.774449 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.774367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68svw\" (UniqueName: \"kubernetes.io/projected/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-kube-api-access-68svw\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:03.775315 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.775298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27pd\" (UniqueName: \"kubernetes.io/projected/82a364ce-5981-412b-9cf0-65da78af29d9-kube-api-access-j27pd\") pod \"node-resolver-2d7rk\" (UID: \"82a364ce-5981-412b-9cf0-65da78af29d9\") " pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.775589 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.775561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfl2j\" (UniqueName: \"kubernetes.io/projected/e1b6be1d-bba4-48f0-900a-a36c80597729-kube-api-access-bfl2j\") pod \"multus-c4lfh\" (UID: \"e1b6be1d-bba4-48f0-900a-a36c80597729\") " pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.775835 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.775814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97pb\" (UniqueName: \"kubernetes.io/projected/e9d0d0c6-75c8-4f1e-8555-9344af2b8d64-kube-api-access-m97pb\") pod \"tuned-qqc4r\" (UID: \"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64\") " pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.775893 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.775827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvzj\" (UniqueName: \"kubernetes.io/projected/b98300df-7e61-45ad-87d0-b0185bb00771-kube-api-access-ncvzj\") pod \"multus-additional-cni-plugins-h2mf6\" (UID: \"b98300df-7e61-45ad-87d0-b0185bb00771\") " pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.866586 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.866538    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crz62"
Sep 11 03:50:03.877354 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.877333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46f4324_94b3_488f_bc24_d5af8a3740ae.slice/crio-df5525066d591f68b880e6d1736ba697f86d4c386d6ba2e4d4cdcc72e57910f8 WatchSource:0}: Error finding container df5525066d591f68b880e6d1736ba697f86d4c386d6ba2e4d4cdcc72e57910f8: Status 404 returned error can't find the container with id df5525066d591f68b880e6d1736ba697f86d4c386d6ba2e4d4cdcc72e57910f8
Sep 11 03:50:03.880501 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.878698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:03.881368 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.881347    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:03.887547 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.887527    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6lvvf"
Sep 11 03:50:03.896299 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.896278    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc800d9_e949_4c5e_9804_1f74c6b507b9.slice/crio-1497792f03beae364d003ffe449e01378e001b1ffa554e569219de13b627eac5 WatchSource:0}: Error finding container 1497792f03beae364d003ffe449e01378e001b1ffa554e569219de13b627eac5: Status 404 returned error can't find the container with id 1497792f03beae364d003ffe449e01378e001b1ffa554e569219de13b627eac5
Sep 11 03:50:03.899720 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.899701    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x"
Sep 11 03:50:03.908458 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.908438    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28be8a53_5cb9_4843_bcaa_5b840e135f08.slice/crio-6662cfce7fba5e44ca8e316efd2d3c18eb0ecadc1d974470cfaf14d2bc59c1aa WatchSource:0}: Error finding container 6662cfce7fba5e44ca8e316efd2d3c18eb0ecadc1d974470cfaf14d2bc59c1aa: Status 404 returned error can't find the container with id 6662cfce7fba5e44ca8e316efd2d3c18eb0ecadc1d974470cfaf14d2bc59c1aa
Sep 11 03:50:03.918354 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.918336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2mf6"
Sep 11 03:50:03.927086 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.927066    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98300df_7e61_45ad_87d0_b0185bb00771.slice/crio-79abc30d80231d3dbc8c007efa909801688ce4321ec319eca73d749ff7006763 WatchSource:0}: Error finding container 79abc30d80231d3dbc8c007efa909801688ce4321ec319eca73d749ff7006763: Status 404 returned error can't find the container with id 79abc30d80231d3dbc8c007efa909801688ce4321ec319eca73d749ff7006763
Sep 11 03:50:03.940459 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.940442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:03.945733 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.945717    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2d7rk"
Sep 11 03:50:03.948624 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.948487    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd262c2_7fea_463a_9c9d_84134976390d.slice/crio-aa232022c16480666b99220049d27d7ae13e9c3bb2cf48173d0f4f2801a24bb4 WatchSource:0}: Error finding container aa232022c16480666b99220049d27d7ae13e9c3bb2cf48173d0f4f2801a24bb4: Status 404 returned error can't find the container with id aa232022c16480666b99220049d27d7ae13e9c3bb2cf48173d0f4f2801a24bb4
Sep 11 03:50:03.952605 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.952589    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqc4r"
Sep 11 03:50:03.954110 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.954090    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a364ce_5981_412b_9cf0_65da78af29d9.slice/crio-4e65062c3010a15d8b60f983f3cf81940d80a17e16c01f93de163b9d64da336b WatchSource:0}: Error finding container 4e65062c3010a15d8b60f983f3cf81940d80a17e16c01f93de163b9d64da336b: Status 404 returned error can't find the container with id 4e65062c3010a15d8b60f983f3cf81940d80a17e16c01f93de163b9d64da336b
Sep 11 03:50:03.958657 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:03.958639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c4lfh"
Sep 11 03:50:03.961563 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.961543    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d0d0c6_75c8_4f1e_8555_9344af2b8d64.slice/crio-6ddc5498140c541abbd9084175b627e42b28bd86523a025afc59c1ea45514248 WatchSource:0}: Error finding container 6ddc5498140c541abbd9084175b627e42b28bd86523a025afc59c1ea45514248: Status 404 returned error can't find the container with id 6ddc5498140c541abbd9084175b627e42b28bd86523a025afc59c1ea45514248
Sep 11 03:50:03.968692 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:03.968668    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b6be1d_bba4_48f0_900a_a36c80597729.slice/crio-3e7b8dcdf82ce55bfc33ffdb5415e52edc723cc3c503fd9fe6fcc753e1ad5fc8 WatchSource:0}: Error finding container 3e7b8dcdf82ce55bfc33ffdb5415e52edc723cc3c503fd9fe6fcc753e1ad5fc8: Status 404 returned error can't find the container with id 3e7b8dcdf82ce55bfc33ffdb5415e52edc723cc3c503fd9fe6fcc753e1ad5fc8
Sep 11 03:50:04.264123 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.263860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:04.264123 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.263929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:04.264123 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.264009    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:04.264123 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.264038    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:04.264123 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.264090    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret podName:37437296-da5a-4a64-8dca-a89fa0263d79 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:05.26406337 +0000 UTC m=+3.130083102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret") pod "global-pull-secret-syncer-frxxs" (UID: "37437296-da5a-4a64-8dca-a89fa0263d79") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:04.264123 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.264112    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs podName:c6f66daa-4ff1-45b4-93d6-cc8de58802b3 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:05.264100958 +0000 UTC m=+3.130120680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs") pod "network-metrics-daemon-k57hw" (UID: "c6f66daa-4ff1-45b4-93d6-cc8de58802b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:04.364504 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.364458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:04.364682 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.364663    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:04.364757 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.364691    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:04.364757 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.364704    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ls7w4 for pod openshift-network-diagnostics/network-check-target-t7pq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:04.364895 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.364765    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4 podName:544a104a-4499-4f6e-a751-77db1ae1aca2 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:05.364745186 +0000 UTC m=+3.230764907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ls7w4" (UniqueName: "kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4") pod "network-check-target-t7pq9" (UID: "544a104a-4499-4f6e-a751-77db1ae1aca2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:04.427571 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.427086    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:04.591253 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.591143    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:45:04 +0000 UTC" deadline="2028-03-20 10:31:21.686166838 +0000 UTC"
Sep 11 03:50:04.591253 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.591207    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13350h41m17.094965352s"
Sep 11 03:50:04.606746 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.606714    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:04.606916 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:04.606870    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:04.644113 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.644057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal" event={"ID":"147e739407ba5f1c7ea961026966fc31","Type":"ContainerStarted","Data":"e1f52a2602738dba0ee2fec8850e76f0c4a0bd5e7967c0eee5096bffad6fee83"}
Sep 11 03:50:04.655015 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.654983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xh7zf" event={"ID":"3cd262c2-7fea-463a-9c9d-84134976390d","Type":"ContainerStarted","Data":"aa232022c16480666b99220049d27d7ae13e9c3bb2cf48173d0f4f2801a24bb4"}
Sep 11 03:50:04.671695 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.671394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" event={"ID":"b98300df-7e61-45ad-87d0-b0185bb00771","Type":"ContainerStarted","Data":"79abc30d80231d3dbc8c007efa909801688ce4321ec319eca73d749ff7006763"}
Sep 11 03:50:04.682877 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.682845    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6lvvf" event={"ID":"cdc800d9-e949-4c5e-9804-1f74c6b507b9","Type":"ContainerStarted","Data":"1497792f03beae364d003ffe449e01378e001b1ffa554e569219de13b627eac5"}
Sep 11 03:50:04.693109 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.693076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"c605a3bf3d7f421a580a4d862bfa36759649ad650a64867a523d942f78d30955"}
Sep 11 03:50:04.706862 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.706833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal" event={"ID":"9e7e89872af809b35925e7ce91f29761","Type":"ContainerStarted","Data":"c4a829fe715213ae66a9bbd25e6316909b7d309fa534738bec44175d01f6e689"}
Sep 11 03:50:04.724042 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.723962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c4lfh" event={"ID":"e1b6be1d-bba4-48f0-900a-a36c80597729","Type":"ContainerStarted","Data":"3e7b8dcdf82ce55bfc33ffdb5415e52edc723cc3c503fd9fe6fcc753e1ad5fc8"}
Sep 11 03:50:04.735602 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.735570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqc4r" event={"ID":"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64","Type":"ContainerStarted","Data":"6ddc5498140c541abbd9084175b627e42b28bd86523a025afc59c1ea45514248"}
Sep 11 03:50:04.747673 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.747594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2d7rk" event={"ID":"82a364ce-5981-412b-9cf0-65da78af29d9","Type":"ContainerStarted","Data":"4e65062c3010a15d8b60f983f3cf81940d80a17e16c01f93de163b9d64da336b"}
Sep 11 03:50:04.778489 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.778457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x" event={"ID":"28be8a53-5cb9-4843-bcaa-5b840e135f08","Type":"ContainerStarted","Data":"6662cfce7fba5e44ca8e316efd2d3c18eb0ecadc1d974470cfaf14d2bc59c1aa"}
Sep 11 03:50:04.790135 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.790106    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crz62" event={"ID":"f46f4324-94b3-488f-bc24-d5af8a3740ae","Type":"ContainerStarted","Data":"df5525066d591f68b880e6d1736ba697f86d4c386d6ba2e4d4cdcc72e57910f8"}
Sep 11 03:50:04.894475 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:04.894400    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:05.272952 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:05.272867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:05.272952 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:05.272936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:05.273182 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.273079    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:05.273182 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.273146    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs podName:c6f66daa-4ff1-45b4-93d6-cc8de58802b3 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:07.273127756 +0000 UTC m=+5.139147490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs") pod "network-metrics-daemon-k57hw" (UID: "c6f66daa-4ff1-45b4-93d6-cc8de58802b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:05.273650 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.273570    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:05.273650 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.273625    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret podName:37437296-da5a-4a64-8dca-a89fa0263d79 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:07.273608973 +0000 UTC m=+5.139628706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret") pod "global-pull-secret-syncer-frxxs" (UID: "37437296-da5a-4a64-8dca-a89fa0263d79") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:05.373490 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:05.373386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:05.373667 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.373583    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:05.373667 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.373607    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:05.373667 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.373622    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ls7w4 for pod openshift-network-diagnostics/network-check-target-t7pq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:05.373827 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.373680    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4 podName:544a104a-4499-4f6e-a751-77db1ae1aca2 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:07.373661485 +0000 UTC m=+5.239681207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ls7w4" (UniqueName: "kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4") pod "network-check-target-t7pq9" (UID: "544a104a-4499-4f6e-a751-77db1ae1aca2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:05.592212 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:05.592063    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:45:04 +0000 UTC" deadline="2028-04-13 01:59:46.618325758 +0000 UTC"
Sep 11 03:50:05.592212 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:05.592103    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13918h9m41.026227535s"
Sep 11 03:50:05.607347 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:05.606624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:05.607347 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.606767    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:05.607347 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:05.607189    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:05.607347 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:05.607296    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:05.750657 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:05.750626    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 03:50:06.608111 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:06.607359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:06.608111 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:06.607535    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:07.293460 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:07.293360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:07.293460 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:07.293448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:07.293691 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.293605    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:07.293691 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.293669    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs podName:c6f66daa-4ff1-45b4-93d6-cc8de58802b3 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:11.293648696 +0000 UTC m=+9.159668415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs") pod "network-metrics-daemon-k57hw" (UID: "c6f66daa-4ff1-45b4-93d6-cc8de58802b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:07.293826 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.293741    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:07.293826 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.293782    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret podName:37437296-da5a-4a64-8dca-a89fa0263d79 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:11.293768302 +0000 UTC m=+9.159788020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret") pod "global-pull-secret-syncer-frxxs" (UID: "37437296-da5a-4a64-8dca-a89fa0263d79") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:07.394125 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:07.394087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:07.394297 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.394279    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:07.394375 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.394299    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:07.394375 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.394313    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ls7w4 for pod openshift-network-diagnostics/network-check-target-t7pq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:07.394375 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.394371    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4 podName:544a104a-4499-4f6e-a751-77db1ae1aca2 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:11.394352706 +0000 UTC m=+9.260372428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ls7w4" (UniqueName: "kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4") pod "network-check-target-t7pq9" (UID: "544a104a-4499-4f6e-a751-77db1ae1aca2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:07.606450 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:07.606373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:07.606632 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.606523    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:07.606947 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:07.606924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:07.607051 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:07.607030    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:08.607609 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:08.607049    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:08.607609 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:08.607192    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:09.607002 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:09.606970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:09.607199 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:09.607108    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:09.607199 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:09.607168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:09.607322 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:09.607286    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:10.608965 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:10.607010    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:10.608965 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:10.607186    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:11.326074 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:11.325247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:11.326074 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:11.325317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:11.326074 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.325500    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:11.326074 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.325572    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs podName:c6f66daa-4ff1-45b4-93d6-cc8de58802b3 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:19.325549152 +0000 UTC m=+17.191568868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs") pod "network-metrics-daemon-k57hw" (UID: "c6f66daa-4ff1-45b4-93d6-cc8de58802b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:11.326074 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.325985    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:11.326074 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.326039    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret podName:37437296-da5a-4a64-8dca-a89fa0263d79 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:19.326022368 +0000 UTC m=+17.192042090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret") pod "global-pull-secret-syncer-frxxs" (UID: "37437296-da5a-4a64-8dca-a89fa0263d79") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:11.426570 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:11.426489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:11.426745 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.426640    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:11.426745 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.426661    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:11.426745 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.426677    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ls7w4 for pod openshift-network-diagnostics/network-check-target-t7pq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:11.426745 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.426735    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4 podName:544a104a-4499-4f6e-a751-77db1ae1aca2 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:19.4267179 +0000 UTC m=+17.292737621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ls7w4" (UniqueName: "kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4") pod "network-check-target-t7pq9" (UID: "544a104a-4499-4f6e-a751-77db1ae1aca2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:11.606876 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:11.606319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:11.606876 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.606465    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:11.606876 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:11.606610    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:11.606876 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:11.606737    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:12.608194 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:12.608161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:12.608797 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:12.608286    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:13.606315 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:13.606277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:13.606522 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:13.606328    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:13.606522 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:13.606415    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:13.606646 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:13.606529    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:14.606088 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:14.606052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:14.606565 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:14.606187    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:15.606628 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:15.606592    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:15.607159 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:15.606719    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:15.607159 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:15.606774    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:15.607159 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:15.606887    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:16.606299 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:16.606228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:16.606454 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:16.606358    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:17.606882 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:17.606844    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:17.606882 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:17.606871    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:17.607364 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:17.606968    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:17.607364 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:17.607045    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:18.606675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:18.606646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:18.606847 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:18.606784    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:19.388839 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:19.388806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:19.389344 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:19.388862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:19.389344 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.388962    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:19.389344 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.388979    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:19.389344 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.389032    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret podName:37437296-da5a-4a64-8dca-a89fa0263d79 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:35.389010105 +0000 UTC m=+33.255029822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret") pod "global-pull-secret-syncer-frxxs" (UID: "37437296-da5a-4a64-8dca-a89fa0263d79") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:19.389344 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.389047    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs podName:c6f66daa-4ff1-45b4-93d6-cc8de58802b3 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:35.389040136 +0000 UTC m=+33.255059850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs") pod "network-metrics-daemon-k57hw" (UID: "c6f66daa-4ff1-45b4-93d6-cc8de58802b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:19.489209 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:19.489178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:19.489377 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.489359    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:19.489451 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.489381    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:19.489451 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.489393    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ls7w4 for pod openshift-network-diagnostics/network-check-target-t7pq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:19.489552 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.489460    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4 podName:544a104a-4499-4f6e-a751-77db1ae1aca2 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:35.489444121 +0000 UTC m=+33.355463852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ls7w4" (UniqueName: "kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4") pod "network-check-target-t7pq9" (UID: "544a104a-4499-4f6e-a751-77db1ae1aca2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:19.606014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:19.605983    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:19.606014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:19.606014    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:19.606240 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.606113    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:19.606314 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:19.606291    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:20.606897 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:20.606863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:20.607321 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:20.606981    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:21.606122 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:21.606090    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:21.606292 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:21.606239    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:21.606369 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:21.606292    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:21.606443 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:21.606375    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:22.606706 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:22.606681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:22.607169 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:22.606763    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:23.606135 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.605980    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:23.606221 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.606036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:23.606274 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:23.606218    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:23.606327 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:23.606301    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:23.834033 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.833999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c4lfh" event={"ID":"e1b6be1d-bba4-48f0-900a-a36c80597729","Type":"ContainerStarted","Data":"73509f23ae7970a15a5fe815d393c3450fa6d3010b15a24a08f5d669a4553e60"}
Sep 11 03:50:23.845306 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.845128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqc4r" event={"ID":"e9d0d0c6-75c8-4f1e-8555-9344af2b8d64","Type":"ContainerStarted","Data":"94cd75714f2e265188fc300eb308234c577c81e117b5b445fbce652f7b6c0e40"}
Sep 11 03:50:23.847966 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.847939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal" event={"ID":"147e739407ba5f1c7ea961026966fc31","Type":"ContainerStarted","Data":"be7483dfbdaf7eb6c0c265af3380e8ef92d62efb5cd106cc2fe7887aa68f604b"}
Sep 11 03:50:23.850908 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.850860    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c4lfh" podStartSLOduration=1.113231581 podStartE2EDuration="20.850842822s" totalImagesPullingTime="19.737611241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:23.850505027 +0000 UTC m=+21.716524763" watchObservedRunningTime="2026-09-11 03:50:23.850842822 +0000 UTC m=+21.716862560"
Sep 11 03:50:23.861207 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.861093    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 03:50:23.862268 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.862039    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a" containerID="58057db5a0e358a9ed5303ad1cbc59a46540687a76d7c57e4ac7eef1b363ae25" exitCode=1
Sep 11 03:50:23.862268 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.862085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"03c75de294062673b79d8840bbc1432461ec1c21d4aedd83747b57554291a4f2"}
Sep 11 03:50:23.862268 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.862156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerDied","Data":"58057db5a0e358a9ed5303ad1cbc59a46540687a76d7c57e4ac7eef1b363ae25"}
Sep 11 03:50:23.862268 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.862209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"b8c2256d05a924cc32bf2239135bcfeedf514f3074ff1a4d40c109ef94df7161"}
Sep 11 03:50:23.864390 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:23.863850    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-122.ec2.internal" podStartSLOduration=19.863833895 podStartE2EDuration="19.863833895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:23.862947061 +0000 UTC m=+21.728966798" watchObservedRunningTime="2026-09-11 03:50:23.863833895 +0000 UTC m=+21.729853632"
Sep 11 03:50:24.606720 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.606690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:24.606918 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:24.606808    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:24.865735 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.865705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2d7rk" event={"ID":"82a364ce-5981-412b-9cf0-65da78af29d9","Type":"ContainerStarted","Data":"ac61709809af8691dcdbe417b8dc017a3da1e9294a24f1de2f9ca4726f5cd875"}
Sep 11 03:50:24.867226 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.867202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x" event={"ID":"28be8a53-5cb9-4843-bcaa-5b840e135f08","Type":"ContainerStarted","Data":"4287b3ad91b79fddc46fa8dc23d5d9c665bff63d1e6ee16b7c55641d18baa82c"}
Sep 11 03:50:24.869333 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.869303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crz62" event={"ID":"f46f4324-94b3-488f-bc24-d5af8a3740ae","Type":"ContainerStarted","Data":"ed03f755761d5500322d63b27ea010662baeda4d2ffc34ed0070b460a12fab74"}
Sep 11 03:50:24.871709 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.871687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xh7zf" event={"ID":"3cd262c2-7fea-463a-9c9d-84134976390d","Type":"ContainerStarted","Data":"0ce3a619f04e041d02afe093040d6a679db592710299f79198cfeb43059787e1"}
Sep 11 03:50:24.873164 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.873142    2566 generic.go:356] "Generic (PLEG): container finished" podID="b98300df-7e61-45ad-87d0-b0185bb00771" containerID="113e27c31df04b1a0b018cde5b9ec94f3ded0075a0efde0c35f366d415899313" exitCode=0
Sep 11 03:50:24.873252 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.873172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" event={"ID":"b98300df-7e61-45ad-87d0-b0185bb00771","Type":"ContainerDied","Data":"113e27c31df04b1a0b018cde5b9ec94f3ded0075a0efde0c35f366d415899313"}
Sep 11 03:50:24.874661 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.874587    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6lvvf" event={"ID":"cdc800d9-e949-4c5e-9804-1f74c6b507b9","Type":"ContainerStarted","Data":"635ea3de49c12a1e7eb73fee71462ec793d3f3a099420dd23e0511ce35652e98"}
Sep 11 03:50:24.877235 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.877218    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 03:50:24.877592 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.877576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"4fe17fb1005730a5b9ec27ca93a89c0610b1d163ac6289574336c072d5c63dd3"}
Sep 11 03:50:24.877658 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.877626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"f54d469d7175ea160d0cce04711e8041d17670d3bc490a742f5faccd3c1b16c7"}
Sep 11 03:50:24.877710 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.877669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"0403754a7a9ea929672dc96d6af1d8ff24d25efad477452875db718290cb053c"}
Sep 11 03:50:24.878897 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.878876    2566 generic.go:356] "Generic (PLEG): container finished" podID="9e7e89872af809b35925e7ce91f29761" containerID="bebe4ee4b8c624461b09c06a661567f5234a4d2d65c7ba0f71c2fd4407931b2b" exitCode=0
Sep 11 03:50:24.878989 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.878950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal" event={"ID":"9e7e89872af809b35925e7ce91f29761","Type":"ContainerDied","Data":"bebe4ee4b8c624461b09c06a661567f5234a4d2d65c7ba0f71c2fd4407931b2b"}
Sep 11 03:50:24.879440 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.879382    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qqc4r" podStartSLOduration=2.468253537 podStartE2EDuration="21.879364833s" totalImagesPullingTime="19.411111296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:23.878759258 +0000 UTC m=+21.744778993" watchObservedRunningTime="2026-09-11 03:50:24.879364833 +0000 UTC m=+22.745384595"
Sep 11 03:50:24.892490 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.892455    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xh7zf" podStartSLOduration=2.471945803 podStartE2EDuration="21.892446502s" totalImagesPullingTime="19.420500699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:24.892432211 +0000 UTC m=+22.758451941" watchObservedRunningTime="2026-09-11 03:50:24.892446502 +0000 UTC m=+22.758466253"
Sep 11 03:50:24.892948 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.892925    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2d7rk" podStartSLOduration=2.496938955 podStartE2EDuration="21.892920272s" totalImagesPullingTime="19.395981317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:24.879677269 +0000 UTC m=+22.745697005" watchObservedRunningTime="2026-09-11 03:50:24.892920272 +0000 UTC m=+22.758940008"
Sep 11 03:50:24.920022 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.919984    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-crz62" podStartSLOduration=2.450592628 podStartE2EDuration="21.919974068s" totalImagesPullingTime="19.46938144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:24.919624365 +0000 UTC m=+22.785644101" watchObservedRunningTime="2026-09-11 03:50:24.919974068 +0000 UTC m=+22.785993804"
Sep 11 03:50:24.974293 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:24.974261    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6lvvf" podStartSLOduration=2.520498061 podStartE2EDuration="21.974248209s" totalImagesPullingTime="19.453750148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:24.943368192 +0000 UTC m=+22.809387927" watchObservedRunningTime="2026-09-11 03:50:24.974248209 +0000 UTC m=+22.840267947"
Sep 11 03:50:25.089810 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.089649    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 03:50:25.606509 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.606479    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:25.606679 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.606489    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:25.606679 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:25.606587    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:25.606679 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:25.606666    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:25.615957 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.615859    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T03:50:25.089790215Z","UUID":"1ea3191b-95b4-4ba5-84c4-ea516b95ae8f","Handler":null,"Name":"","Endpoint":""}
Sep 11 03:50:25.617705 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.617656    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 03:50:25.617705 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.617688    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 03:50:25.883233 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.883139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal" event={"ID":"9e7e89872af809b35925e7ce91f29761","Type":"ContainerStarted","Data":"bb312b4cbcf54f3011100339fb97b3d904c5b3b15f3566fd061b669ad535efb4"}
Sep 11 03:50:25.885397 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.885371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x" event={"ID":"28be8a53-5cb9-4843-bcaa-5b840e135f08","Type":"ContainerStarted","Data":"cfbae4110e67f3313f58c1781f2ec9223f2452e3989cbdd649cec1f1f82ae6be"}
Sep 11 03:50:25.897867 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:25.897817    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-122.ec2.internal" podStartSLOduration=21.897800648 podStartE2EDuration="21.897800648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:25.897547559 +0000 UTC m=+23.763567296" watchObservedRunningTime="2026-09-11 03:50:25.897800648 +0000 UTC m=+23.763820387"
Sep 11 03:50:26.347135 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.347105    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:26.347725 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.347707    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:26.606850 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.606783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:26.606970 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:26.606888    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:26.889093 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.889024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x" event={"ID":"28be8a53-5cb9-4843-bcaa-5b840e135f08","Type":"ContainerStarted","Data":"e12f02a8db8fc7587785f8900c4c82ef5f243061efdad957cbb6efd11181110d"}
Sep 11 03:50:26.891952 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.891932    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 03:50:26.892326 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.892304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"21932c7a8187fbdacc1ee14d5ef7d7ccd60643f26e726fb773cc9214bef10f03"}
Sep 11 03:50:26.892614 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.892597    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:26.893070 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.893050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xh7zf"
Sep 11 03:50:26.947381 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:26.947343    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zm7x" podStartSLOduration=2.74876729 podStartE2EDuration="23.947330657s" totalImagesPullingTime="21.198563367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:26.920285745 +0000 UTC m=+24.786305481" watchObservedRunningTime="2026-09-11 03:50:26.947330657 +0000 UTC m=+24.813350438"
Sep 11 03:50:27.606522 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:27.606490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:27.606522 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:27.606521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:27.606743 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:27.606601    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:27.606743 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:27.606700    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:28.606436 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:28.606198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:28.606880 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:28.606503    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:28.900514 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:28.900449    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 03:50:28.900924 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:28.900892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"df3754e99baf8f5c064e5d9b8b04469d9ca0c8bdef305cfeac73b141d2bc1a6f"}
Sep 11 03:50:28.901337 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:28.901312    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:28.901458 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:28.901360    2566 scope.go:117] "RemoveContainer" containerID="58057db5a0e358a9ed5303ad1cbc59a46540687a76d7c57e4ac7eef1b363ae25"
Sep 11 03:50:28.917847 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:28.917825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:29.606008 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:29.605971    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:29.606008 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:29.606000    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:29.606212 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:29.606092    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:29.606303 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:29.606262    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:30.427433 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.427391    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-frxxs"]
Sep 11 03:50:30.428083 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.427547    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:30.428083 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:30.427638    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:30.430845 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.430803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7pq9"]
Sep 11 03:50:30.430968 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.430911    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:30.431036 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:30.431012    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:30.431538 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.431507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k57hw"]
Sep 11 03:50:30.431642 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.431621    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:30.431769 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:30.431749    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:30.907512 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.907478    2566 generic.go:356] "Generic (PLEG): container finished" podID="b98300df-7e61-45ad-87d0-b0185bb00771" containerID="a8f4b5d48a3a38a013fa45b3db480149f2782b5fd7d315a1745d642b7e416067" exitCode=0
Sep 11 03:50:30.907655 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.907558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" event={"ID":"b98300df-7e61-45ad-87d0-b0185bb00771","Type":"ContainerDied","Data":"a8f4b5d48a3a38a013fa45b3db480149f2782b5fd7d315a1745d642b7e416067"}
Sep 11 03:50:30.911643 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.911625    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 03:50:30.911992 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.911969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" event={"ID":"d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a","Type":"ContainerStarted","Data":"944a328fb377a51968e4efb7439d0a02f5f2953da966616e6fd17b84868147b4"}
Sep 11 03:50:30.912231 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.912210    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:30.912339 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.912279    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:30.927298 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.927280    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:50:30.951135 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:30.951093    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh" podStartSLOduration=8.432599079 podStartE2EDuration="27.951080155s" totalImagesPullingTime="19.518481076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:30.951050056 +0000 UTC m=+28.817069792" watchObservedRunningTime="2026-09-11 03:50:30.951080155 +0000 UTC m=+28.817099872"
Sep 11 03:50:31.606176 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:31.606152    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:31.606575 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:31.606154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:31.606575 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:31.606276    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:31.606575 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:31.606331    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:32.607592 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:32.607520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:32.607960 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:32.607628    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:32.919819 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:32.919788    2566 generic.go:356] "Generic (PLEG): container finished" podID="b98300df-7e61-45ad-87d0-b0185bb00771" containerID="c40e18bbba7128f8e6a12b231df750f263f17d0faf4ec762bfb874095ec2cef5" exitCode=0
Sep 11 03:50:32.919929 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:32.919864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" event={"ID":"b98300df-7e61-45ad-87d0-b0185bb00771","Type":"ContainerDied","Data":"c40e18bbba7128f8e6a12b231df750f263f17d0faf4ec762bfb874095ec2cef5"}
Sep 11 03:50:33.606084 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:33.606058    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:33.606084 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:33.606085    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:33.606267 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:33.606167    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:33.606267 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:33.606234    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:33.924000 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:33.923937    2566 generic.go:356] "Generic (PLEG): container finished" podID="b98300df-7e61-45ad-87d0-b0185bb00771" containerID="9296c562157ac6f0bd5302c657ff8a1ab1ca175205de4710e950251083120d99" exitCode=0
Sep 11 03:50:33.924000 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:33.923991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" event={"ID":"b98300df-7e61-45ad-87d0-b0185bb00771","Type":"ContainerDied","Data":"9296c562157ac6f0bd5302c657ff8a1ab1ca175205de4710e950251083120d99"}
Sep 11 03:50:34.606227 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:34.606193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:34.606433 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:34.606312    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frxxs" podUID="37437296-da5a-4a64-8dca-a89fa0263d79"
Sep 11 03:50:35.411305 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:35.411268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:35.411767 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:35.411334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:35.411767 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.411442    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:35.411767 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.411474    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:35.411767 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.411525    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret podName:37437296-da5a-4a64-8dca-a89fa0263d79 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:07.411501733 +0000 UTC m=+65.277521447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret") pod "global-pull-secret-syncer-frxxs" (UID: "37437296-da5a-4a64-8dca-a89fa0263d79") : object "kube-system"/"original-pull-secret" not registered
Sep 11 03:50:35.411767 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.411546    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs podName:c6f66daa-4ff1-45b4-93d6-cc8de58802b3 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:07.411535147 +0000 UTC m=+65.277554863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs") pod "network-metrics-daemon-k57hw" (UID: "c6f66daa-4ff1-45b4-93d6-cc8de58802b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 03:50:35.512515 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:35.512488    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:35.512677 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.512626    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 03:50:35.512677 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.512648    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 03:50:35.512677 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.512657    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ls7w4 for pod openshift-network-diagnostics/network-check-target-t7pq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:35.512819 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.512701    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4 podName:544a104a-4499-4f6e-a751-77db1ae1aca2 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:07.512687995 +0000 UTC m=+65.378707710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ls7w4" (UniqueName: "kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4") pod "network-check-target-t7pq9" (UID: "544a104a-4499-4f6e-a751-77db1ae1aca2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 03:50:35.605969 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:35.605942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:35.606144 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.606050    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k57hw" podUID="c6f66daa-4ff1-45b4-93d6-cc8de58802b3"
Sep 11 03:50:35.606144 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:35.606111    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:35.606264 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:35.606215    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7pq9" podUID="544a104a-4499-4f6e-a751-77db1ae1aca2"
Sep 11 03:50:36.417518 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.417454    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-122.ec2.internal" event="NodeReady"
Sep 11 03:50:36.417840 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.417561    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 03:50:36.449291 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.449262    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79fc457b8f-7nqkf"]
Sep 11 03:50:36.452346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.452329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.454662 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.454639    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 03:50:36.455055 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.454797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 03:50:36.455055 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.454861    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 03:50:36.455055 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.454861    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qn2bv\""
Sep 11 03:50:36.459695 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.459644    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 03:50:36.459968 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.459930    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-btpdt"]
Sep 11 03:50:36.463304 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.463288    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2trdb"]
Sep 11 03:50:36.463439 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.463421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-btpdt"
Sep 11 03:50:36.466009 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.465975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 03:50:36.466180 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.466029    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 03:50:36.466180 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.466056    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g2lkd\""
Sep 11 03:50:36.466359 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.466290    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 03:50:36.466933 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.466915    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79fc457b8f-7nqkf"]
Sep 11 03:50:36.467026 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.467009    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.468991 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.468972    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 03:50:36.469276 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.469261    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hznd4\""
Sep 11 03:50:36.469338 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.469267    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 03:50:36.474610 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.474585    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-btpdt"]
Sep 11 03:50:36.474702 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.474635    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2trdb"]
Sep 11 03:50:36.606915 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.606887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:50:36.609535 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.609515    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:50:36.617710 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3f4a25-3d98-48b6-9026-045c9e3df008-tmp-dir\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.617808 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65a48c04-6e08-42ab-ad40-11acc7435662-ca-trust-extracted\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.617808 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3f4a25-3d98-48b6-9026-045c9e3df008-config-volume\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.617808 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-trusted-ca\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.617808 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-bound-sa-token\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.618014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnvc\" (UniqueName: \"kubernetes.io/projected/4eeec831-824a-4089-87ff-a203bf1d9a87-kube-api-access-qhnvc\") pod \"ingress-canary-btpdt\" (UID: \"4eeec831-824a-4089-87ff-a203bf1d9a87\") " pod="openshift-ingress-canary/ingress-canary-btpdt"
Sep 11 03:50:36.618014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4p6\" (UniqueName: \"kubernetes.io/projected/6e3f4a25-3d98-48b6-9026-045c9e3df008-kube-api-access-mg4p6\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.618014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-image-registry-private-configuration\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.618014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.618014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-installation-pull-secrets\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.618014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.617999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5bgw\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-kube-api-access-z5bgw\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.618307 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.618025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4eeec831-824a-4089-87ff-a203bf1d9a87-cert\") pod \"ingress-canary-btpdt\" (UID: \"4eeec831-824a-4089-87ff-a203bf1d9a87\") " pod="openshift-ingress-canary/ingress-canary-btpdt"
Sep 11 03:50:36.618307 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.618047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.618307 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.618064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-registry-certificates\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719145 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3f4a25-3d98-48b6-9026-045c9e3df008-tmp-dir\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.719260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65a48c04-6e08-42ab-ad40-11acc7435662-ca-trust-extracted\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3f4a25-3d98-48b6-9026-045c9e3df008-config-volume\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.719260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-trusted-ca\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-bound-sa-token\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnvc\" (UniqueName: \"kubernetes.io/projected/4eeec831-824a-4089-87ff-a203bf1d9a87-kube-api-access-qhnvc\") pod \"ingress-canary-btpdt\" (UID: \"4eeec831-824a-4089-87ff-a203bf1d9a87\") " pod="openshift-ingress-canary/ingress-canary-btpdt"
Sep 11 03:50:36.719260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4p6\" (UniqueName: \"kubernetes.io/projected/6e3f4a25-3d98-48b6-9026-045c9e3df008-kube-api-access-mg4p6\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-image-registry-private-configuration\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-installation-pull-secrets\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5bgw\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-kube-api-access-z5bgw\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4eeec831-824a-4089-87ff-a203bf1d9a87-cert\") pod \"ingress-canary-btpdt\" (UID: \"4eeec831-824a-4089-87ff-a203bf1d9a87\") " pod="openshift-ingress-canary/ingress-canary-btpdt"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-registry-certificates\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3f4a25-3d98-48b6-9026-045c9e3df008-tmp-dir\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.719582 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:36.719522    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:50:36.720015 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:36.719591    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fc457b8f-7nqkf: secret "image-registry-tls" not found
Sep 11 03:50:36.720015 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:36.719645    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls podName:65a48c04-6e08-42ab-ad40-11acc7435662 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:37.219629496 +0000 UTC m=+35.085649231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls") pod "image-registry-79fc457b8f-7nqkf" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662") : secret "image-registry-tls" not found
Sep 11 03:50:36.720015 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:36.719651    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:50:36.720015 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:36.719699    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls podName:6e3f4a25-3d98-48b6-9026-045c9e3df008 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:37.219682855 +0000 UTC m=+35.085702583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls") pod "dns-default-2trdb" (UID: "6e3f4a25-3d98-48b6-9026-045c9e3df008") : secret "dns-default-metrics-tls" not found
Sep 11 03:50:36.720015 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65a48c04-6e08-42ab-ad40-11acc7435662-ca-trust-extracted\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.720015 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.719881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3f4a25-3d98-48b6-9026-045c9e3df008-config-volume\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.720312 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.720064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-registry-certificates\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.720312 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.720217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-trusted-ca\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.723471 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.723453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4eeec831-824a-4089-87ff-a203bf1d9a87-cert\") pod \"ingress-canary-btpdt\" (UID: \"4eeec831-824a-4089-87ff-a203bf1d9a87\") " pod="openshift-ingress-canary/ingress-canary-btpdt"
Sep 11 03:50:36.723535 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.723470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-image-registry-private-configuration\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.723535 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.723470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-installation-pull-secrets\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.730313 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.728174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-bound-sa-token\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.730313 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.728684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnvc\" (UniqueName: \"kubernetes.io/projected/4eeec831-824a-4089-87ff-a203bf1d9a87-kube-api-access-qhnvc\") pod \"ingress-canary-btpdt\" (UID: \"4eeec831-824a-4089-87ff-a203bf1d9a87\") " pod="openshift-ingress-canary/ingress-canary-btpdt"
Sep 11 03:50:36.730313 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.728984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4p6\" (UniqueName: \"kubernetes.io/projected/6e3f4a25-3d98-48b6-9026-045c9e3df008-kube-api-access-mg4p6\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:36.730313 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.729251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5bgw\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-kube-api-access-z5bgw\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:36.776117 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:36.776092    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-btpdt"
Sep 11 03:50:37.222782 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.222749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:37.222969 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:37.222927    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:50:37.223042 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.222982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:37.223042 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:37.223025    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls podName:6e3f4a25-3d98-48b6-9026-045c9e3df008 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:38.223002896 +0000 UTC m=+36.089022633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls") pod "dns-default-2trdb" (UID: "6e3f4a25-3d98-48b6-9026-045c9e3df008") : secret "dns-default-metrics-tls" not found
Sep 11 03:50:37.223173 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:37.223084    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:50:37.223173 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:37.223099    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fc457b8f-7nqkf: secret "image-registry-tls" not found
Sep 11 03:50:37.223173 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:37.223158    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls podName:65a48c04-6e08-42ab-ad40-11acc7435662 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:38.223142233 +0000 UTC m=+36.089161950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls") pod "image-registry-79fc457b8f-7nqkf" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662") : secret "image-registry-tls" not found
Sep 11 03:50:37.229129 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.229107    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn"]
Sep 11 03:50:37.230913 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.230898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn"
Sep 11 03:50:37.233353 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.233325    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 03:50:37.233493 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.233447    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mhc4p\""
Sep 11 03:50:37.233546 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.233496    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 03:50:37.243010 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.242990    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn"]
Sep 11 03:50:37.324174 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.324148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwzlz\" (UniqueName: \"kubernetes.io/projected/84d18f9d-c86c-45fe-9fd9-53883ccc5eb7-kube-api-access-hwzlz\") pod \"migrator-8b4d697c5-6bgsn\" (UID: \"84d18f9d-c86c-45fe-9fd9-53883ccc5eb7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn"
Sep 11 03:50:37.424699 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.424671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzlz\" (UniqueName: \"kubernetes.io/projected/84d18f9d-c86c-45fe-9fd9-53883ccc5eb7-kube-api-access-hwzlz\") pod \"migrator-8b4d697c5-6bgsn\" (UID: \"84d18f9d-c86c-45fe-9fd9-53883ccc5eb7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn"
Sep 11 03:50:37.433653 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.433626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzlz\" (UniqueName: \"kubernetes.io/projected/84d18f9d-c86c-45fe-9fd9-53883ccc5eb7-kube-api-access-hwzlz\") pod \"migrator-8b4d697c5-6bgsn\" (UID: \"84d18f9d-c86c-45fe-9fd9-53883ccc5eb7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn"
Sep 11 03:50:37.540685 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.540616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn"
Sep 11 03:50:37.606780 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.606751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:50:37.606886 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.606785    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:50:37.609780 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.609760    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:50:37.609893 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.609804    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:50:37.609893 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.609871    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-58qzc\""
Sep 11 03:50:37.609893 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.609885    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-snbvm\""
Sep 11 03:50:37.610075 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:37.609898    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:50:38.231014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.230984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:38.231171 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.231027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:38.231171 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:38.231130    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:50:38.231171 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:38.231135    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:50:38.231171 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:38.231153    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fc457b8f-7nqkf: secret "image-registry-tls" not found
Sep 11 03:50:38.231292 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:38.231179    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls podName:6e3f4a25-3d98-48b6-9026-045c9e3df008 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:40.231166319 +0000 UTC m=+38.097186034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls") pod "dns-default-2trdb" (UID: "6e3f4a25-3d98-48b6-9026-045c9e3df008") : secret "dns-default-metrics-tls" not found
Sep 11 03:50:38.231292 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:38.231193    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls podName:65a48c04-6e08-42ab-ad40-11acc7435662 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:40.231186319 +0000 UTC m=+38.097206032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls") pod "image-registry-79fc457b8f-7nqkf" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662") : secret "image-registry-tls" not found
Sep 11 03:50:38.629169 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.629105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn"]
Sep 11 03:50:38.630351 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.630040    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-btpdt"]
Sep 11 03:50:38.633219 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:38.633198    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d18f9d_c86c_45fe_9fd9_53883ccc5eb7.slice/crio-e299d8fe6de440f92355ef690f41fc45a34d16d31b2f1191f72b13cbd4792831 WatchSource:0}: Error finding container e299d8fe6de440f92355ef690f41fc45a34d16d31b2f1191f72b13cbd4792831: Status 404 returned error can't find the container with id e299d8fe6de440f92355ef690f41fc45a34d16d31b2f1191f72b13cbd4792831
Sep 11 03:50:38.633578 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:38.633559    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eeec831_824a_4089_87ff_a203bf1d9a87.slice/crio-22355de45f292f8e739407e0b600fb19c4152dcfe5b31581eff5361b34aaccba WatchSource:0}: Error finding container 22355de45f292f8e739407e0b600fb19c4152dcfe5b31581eff5361b34aaccba: Status 404 returned error can't find the container with id 22355de45f292f8e739407e0b600fb19c4152dcfe5b31581eff5361b34aaccba
Sep 11 03:50:38.936911 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.936884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn" event={"ID":"84d18f9d-c86c-45fe-9fd9-53883ccc5eb7","Type":"ContainerStarted","Data":"e299d8fe6de440f92355ef690f41fc45a34d16d31b2f1191f72b13cbd4792831"}
Sep 11 03:50:38.937891 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.937863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-btpdt" event={"ID":"4eeec831-824a-4089-87ff-a203bf1d9a87","Type":"ContainerStarted","Data":"22355de45f292f8e739407e0b600fb19c4152dcfe5b31581eff5361b34aaccba"}
Sep 11 03:50:38.940417 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.940382    2566 generic.go:356] "Generic (PLEG): container finished" podID="b98300df-7e61-45ad-87d0-b0185bb00771" containerID="878c8899d0013869826388227ccc5bc447cdbb0318da1251aa68749023b33463" exitCode=0
Sep 11 03:50:38.940493 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.940426    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" event={"ID":"b98300df-7e61-45ad-87d0-b0185bb00771","Type":"ContainerDied","Data":"878c8899d0013869826388227ccc5bc447cdbb0318da1251aa68749023b33463"}
Sep 11 03:50:38.962938 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.962479    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-k27lb"]
Sep 11 03:50:38.995485 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.995459    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-k27lb"]
Sep 11 03:50:38.995600 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.995580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:38.998283 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.998264    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 03:50:38.998682 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.998495    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 03:50:38.998682 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.998518    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 03:50:38.998682 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.998540    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 03:50:38.998682 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:38.998592    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-x8pkm\""
Sep 11 03:50:39.138883 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.138828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tsw\" (UniqueName: \"kubernetes.io/projected/e6349803-26b7-4d24-ad29-7e4bbeb16953-kube-api-access-97tsw\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.138883 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.138867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6349803-26b7-4d24-ad29-7e4bbeb16953-signing-key\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.139015 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.138887    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6349803-26b7-4d24-ad29-7e4bbeb16953-signing-cabundle\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.239545 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.239509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tsw\" (UniqueName: \"kubernetes.io/projected/e6349803-26b7-4d24-ad29-7e4bbeb16953-kube-api-access-97tsw\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.239691 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.239583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6349803-26b7-4d24-ad29-7e4bbeb16953-signing-key\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.239691 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.239615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6349803-26b7-4d24-ad29-7e4bbeb16953-signing-cabundle\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.240365 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.240334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6349803-26b7-4d24-ad29-7e4bbeb16953-signing-cabundle\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.243022 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.243000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6349803-26b7-4d24-ad29-7e4bbeb16953-signing-key\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.248186 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.248164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tsw\" (UniqueName: \"kubernetes.io/projected/e6349803-26b7-4d24-ad29-7e4bbeb16953-kube-api-access-97tsw\") pod \"service-ca-568869fdc5-k27lb\" (UID: \"e6349803-26b7-4d24-ad29-7e4bbeb16953\") " pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.306056 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.306030    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-k27lb"
Sep 11 03:50:39.615526 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.615454    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-k27lb"]
Sep 11 03:50:39.622071 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:39.622042    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6349803_26b7_4d24_ad29_7e4bbeb16953.slice/crio-ff466884163648596557089a103d37218c612555aecf48ed6ed4943b9583b057 WatchSource:0}: Error finding container ff466884163648596557089a103d37218c612555aecf48ed6ed4943b9583b057: Status 404 returned error can't find the container with id ff466884163648596557089a103d37218c612555aecf48ed6ed4943b9583b057
Sep 11 03:50:39.945107 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.945067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-k27lb" event={"ID":"e6349803-26b7-4d24-ad29-7e4bbeb16953","Type":"ContainerStarted","Data":"ff466884163648596557089a103d37218c612555aecf48ed6ed4943b9583b057"}
Sep 11 03:50:39.948854 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.948827    2566 generic.go:356] "Generic (PLEG): container finished" podID="b98300df-7e61-45ad-87d0-b0185bb00771" containerID="3afb815b073611ee0d47be4d32d9160ee3840558ffa4bc2f244f7490b3a2153f" exitCode=0
Sep 11 03:50:39.949009 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:39.948871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" event={"ID":"b98300df-7e61-45ad-87d0-b0185bb00771","Type":"ContainerDied","Data":"3afb815b073611ee0d47be4d32d9160ee3840558ffa4bc2f244f7490b3a2153f"}
Sep 11 03:50:40.245870 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:40.245804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:40.245972 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:40.245900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:40.245972 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:40.245935    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:50:40.246086 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:40.245996    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls podName:6e3f4a25-3d98-48b6-9026-045c9e3df008 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:44.245976785 +0000 UTC m=+42.111996513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls") pod "dns-default-2trdb" (UID: "6e3f4a25-3d98-48b6-9026-045c9e3df008") : secret "dns-default-metrics-tls" not found
Sep 11 03:50:40.246086 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:40.246007    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:50:40.246086 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:40.246019    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fc457b8f-7nqkf: secret "image-registry-tls" not found
Sep 11 03:50:40.246086 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:40.246062    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls podName:65a48c04-6e08-42ab-ad40-11acc7435662 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:44.246047106 +0000 UTC m=+42.112066840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls") pod "image-registry-79fc457b8f-7nqkf" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662") : secret "image-registry-tls" not found
Sep 11 03:50:40.956741 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:40.956708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" event={"ID":"b98300df-7e61-45ad-87d0-b0185bb00771","Type":"ContainerStarted","Data":"3f244aaaa498f1e2c8b8ecb124df2eb936df64a4ed8d40e7a4c86bce8162c968"}
Sep 11 03:50:40.958707 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:40.958675    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn" event={"ID":"84d18f9d-c86c-45fe-9fd9-53883ccc5eb7","Type":"ContainerStarted","Data":"345d0f036c9d3d5436c15db701ba35e74b5613464dd57486e46125c9d876e727"}
Sep 11 03:50:41.002791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:41.002730    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h2mf6" podStartSLOduration=6.574652883 podStartE2EDuration="38.002707708s" totalImagesPullingTime="31.428054825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:40.985487974 +0000 UTC m=+38.851507710" watchObservedRunningTime="2026-09-11 03:50:41.002707708 +0000 UTC m=+38.868727445"
Sep 11 03:50:41.963196 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:41.963157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-k27lb" event={"ID":"e6349803-26b7-4d24-ad29-7e4bbeb16953","Type":"ContainerStarted","Data":"92752e77376d83ee1dc549bd3e0234bff8843af8b4c88bd238b61d6d3de06e3e"}
Sep 11 03:50:41.965053 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:41.965018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn" event={"ID":"84d18f9d-c86c-45fe-9fd9-53883ccc5eb7","Type":"ContainerStarted","Data":"1da8978ccbf94822baf54f74172db5eb662ac5facf7ebc59eda5a09aae252224"}
Sep 11 03:50:41.966614 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:41.966584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-btpdt" event={"ID":"4eeec831-824a-4089-87ff-a203bf1d9a87","Type":"ContainerStarted","Data":"897d9edebe5bd0f4319b79a6900eca8f0914c406d4127199e140a752e399b423"}
Sep 11 03:50:41.982025 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:41.981978    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-k27lb" podStartSLOduration=1.836144695 podStartE2EDuration="3.981964189s" totalImagesPullingTime="2.145819494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:41.980449393 +0000 UTC m=+39.846469127" watchObservedRunningTime="2026-09-11 03:50:41.981964189 +0000 UTC m=+39.847983959"
Sep 11 03:50:41.996542 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:41.996498    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-btpdt" podStartSLOduration=2.864234184 podStartE2EDuration="5.996482807s" totalImagesPullingTime="3.132248623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:41.996121252 +0000 UTC m=+39.862140990" watchObservedRunningTime="2026-09-11 03:50:41.996482807 +0000 UTC m=+39.862502543"
Sep 11 03:50:42.012014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:42.011937    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6bgsn" podStartSLOduration=3.517487278 podStartE2EDuration="5.011920125s" totalImagesPullingTime="1.494432847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:42.011100191 +0000 UTC m=+39.877119939" watchObservedRunningTime="2026-09-11 03:50:42.011920125 +0000 UTC m=+39.877939859"
Sep 11 03:50:44.278688 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:44.278654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:44.279148 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:44.278736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:44.279148 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:44.278830    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 03:50:44.279148 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:44.278890    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls podName:6e3f4a25-3d98-48b6-9026-045c9e3df008 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:52.278876158 +0000 UTC m=+50.144895876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls") pod "dns-default-2trdb" (UID: "6e3f4a25-3d98-48b6-9026-045c9e3df008") : secret "dns-default-metrics-tls" not found
Sep 11 03:50:44.279148 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:44.278834    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 03:50:44.279148 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:44.278938    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fc457b8f-7nqkf: secret "image-registry-tls" not found
Sep 11 03:50:44.279148 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:50:44.278982    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls podName:65a48c04-6e08-42ab-ad40-11acc7435662 nodeName:}" failed. No retries permitted until 2026-09-11 03:50:52.278968238 +0000 UTC m=+50.144987966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls") pod "image-registry-79fc457b8f-7nqkf" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662") : secret "image-registry-tls" not found
Sep 11 03:50:52.337651 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:52.337619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:52.338148 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:52.337686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:52.340951 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:52.340922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"image-registry-79fc457b8f-7nqkf\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") " pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:52.340951 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:52.340933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e3f4a25-3d98-48b6-9026-045c9e3df008-metrics-tls\") pod \"dns-default-2trdb\" (UID: \"6e3f4a25-3d98-48b6-9026-045c9e3df008\") " pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:52.363916 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:52.363892    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:52.383053 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:52.383031    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:52.688081 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:52.688007    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79fc457b8f-7nqkf"]
Sep 11 03:50:52.693206 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:52.693180    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a48c04_6e08_42ab_ad40_11acc7435662.slice/crio-e68b1c827367ab7c433403ed634aeb172d49a7ac71f99481d2e1ecc14ff52daa WatchSource:0}: Error finding container e68b1c827367ab7c433403ed634aeb172d49a7ac71f99481d2e1ecc14ff52daa: Status 404 returned error can't find the container with id e68b1c827367ab7c433403ed634aeb172d49a7ac71f99481d2e1ecc14ff52daa
Sep 11 03:50:52.708823 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:52.708803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2trdb"]
Sep 11 03:50:52.711043 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:50:52.711020    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3f4a25_3d98_48b6_9026_045c9e3df008.slice/crio-75ea3196d5800caf457f19712595ef4b72ff95ad8c146802f1f83b7377ff85f5 WatchSource:0}: Error finding container 75ea3196d5800caf457f19712595ef4b72ff95ad8c146802f1f83b7377ff85f5: Status 404 returned error can't find the container with id 75ea3196d5800caf457f19712595ef4b72ff95ad8c146802f1f83b7377ff85f5
Sep 11 03:50:53.001060 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:53.001028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf" event={"ID":"65a48c04-6e08-42ab-ad40-11acc7435662","Type":"ContainerStarted","Data":"bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140"}
Sep 11 03:50:53.001233 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:53.001078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf" event={"ID":"65a48c04-6e08-42ab-ad40-11acc7435662","Type":"ContainerStarted","Data":"e68b1c827367ab7c433403ed634aeb172d49a7ac71f99481d2e1ecc14ff52daa"}
Sep 11 03:50:53.001233 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:53.001102    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:50:53.002330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:53.002305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2trdb" event={"ID":"6e3f4a25-3d98-48b6-9026-045c9e3df008","Type":"ContainerStarted","Data":"75ea3196d5800caf457f19712595ef4b72ff95ad8c146802f1f83b7377ff85f5"}
Sep 11 03:50:53.019643 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:53.019602    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf" podStartSLOduration=51.019586603 podStartE2EDuration="51.019586603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:53.018712843 +0000 UTC m=+50.884732571" watchObservedRunningTime="2026-09-11 03:50:53.019586603 +0000 UTC m=+50.885606339"
Sep 11 03:50:55.010938 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:55.010907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2trdb" event={"ID":"6e3f4a25-3d98-48b6-9026-045c9e3df008","Type":"ContainerStarted","Data":"acc6c3b9331d56a3514b290a1fdf99f7306f487e433828f611f5bdfb56ceb866"}
Sep 11 03:50:55.011254 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:55.010961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2trdb" event={"ID":"6e3f4a25-3d98-48b6-9026-045c9e3df008","Type":"ContainerStarted","Data":"b7d4a1de4917badf03262a97f7d8c119c1a1db7b48cc8bada563eaba51d10bec"}
Sep 11 03:50:55.011254 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:55.011000    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2trdb"
Sep 11 03:50:55.028345 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:50:55.028300    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2trdb" podStartSLOduration=17.237047797 podStartE2EDuration="19.028285898s" totalImagesPullingTime="1.791238101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:50:55.026752319 +0000 UTC m=+52.892772057" watchObservedRunningTime="2026-09-11 03:50:55.028285898 +0000 UTC m=+52.894305690"
Sep 11 03:51:01.378839 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.378805    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"]
Sep 11 03:51:01.408780 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.408747    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"]
Sep 11 03:51:01.408953 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.408870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.411953 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.411922    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 03:51:01.412096 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.412023    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 03:51:01.412858 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.412833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 03:51:01.412976 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.412906    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 03:51:01.422040 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.422017    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"]
Sep 11 03:51:01.422125 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.422057    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"]
Sep 11 03:51:01.422185 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.422128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.424466 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.424448    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 03:51:01.424636 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.424611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 03:51:01.424732 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.424659    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 03:51:01.424732 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.424702    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 03:51:01.503862 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.503834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff00cced-0f7d-437b-a2f1-ccb730276edb-klusterlet-config\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.504001 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.503867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wxr\" (UniqueName: \"kubernetes.io/projected/ff00cced-0f7d-437b-a2f1-ccb730276edb-kube-api-access-f2wxr\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.504001 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.503906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff00cced-0f7d-437b-a2f1-ccb730276edb-tmp\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.505277 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.505251    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79fc457b8f-7nqkf"]
Sep 11 03:51:01.604679 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.604652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-ca\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.604846 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.604717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.604846 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.604786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.604970 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.604878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff00cced-0f7d-437b-a2f1-ccb730276edb-klusterlet-config\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.604970 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.604909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wxr\" (UniqueName: \"kubernetes.io/projected/ff00cced-0f7d-437b-a2f1-ccb730276edb-kube-api-access-f2wxr\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.604970 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.604934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-hub\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.605109 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.604983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.605109 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.605014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff00cced-0f7d-437b-a2f1-ccb730276edb-tmp\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.605109 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.605037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzzjt\" (UniqueName: \"kubernetes.io/projected/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-kube-api-access-jzzjt\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.605391 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.605367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff00cced-0f7d-437b-a2f1-ccb730276edb-tmp\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.607180 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.607160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ff00cced-0f7d-437b-a2f1-ccb730276edb-klusterlet-config\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.609030 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.609006    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86d879bdf9-p9lqk"]
Sep 11 03:51:01.615339 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.615323    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.629565 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.629517    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86d879bdf9-p9lqk"]
Sep 11 03:51:01.631206 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.631186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wxr\" (UniqueName: \"kubernetes.io/projected/ff00cced-0f7d-437b-a2f1-ccb730276edb-kube-api-access-f2wxr\") pod \"klusterlet-addon-workmgr-669db9d7c6-7sntl\" (UID: \"ff00cced-0f7d-437b-a2f1-ccb730276edb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.705823 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.705799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.705953 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.705834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.705953 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.705902    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-hub\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.705953 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.705950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.706095 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.705971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzzjt\" (UniqueName: \"kubernetes.io/projected/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-kube-api-access-jzzjt\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.706095 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.705994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-ca\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.706694 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.706668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.708263 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.708240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-ca\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.708367 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.708340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.708451 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.708371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.708516 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.708467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-hub\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.718504 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.718481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:01.719928 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.719909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzzjt\" (UniqueName: \"kubernetes.io/projected/7d8f8531-c5fa-43e7-a72b-552d78ffde7c-kube-api-access-jzzjt\") pod \"cluster-proxy-proxy-agent-6f6f9966b6-bplrp\" (UID: \"7d8f8531-c5fa-43e7-a72b-552d78ffde7c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.740236 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.740198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"
Sep 11 03:51:01.807060 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.807031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f70e391d-9436-4989-b9ed-992782a0b750-ca-trust-extracted\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.807163 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.807096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-bound-sa-token\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.807163 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.807134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f70e391d-9436-4989-b9ed-992782a0b750-trusted-ca\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.807253 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.807179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f70e391d-9436-4989-b9ed-992782a0b750-installation-pull-secrets\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.807253 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.807226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-registry-tls\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.807328 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.807251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f70e391d-9436-4989-b9ed-992782a0b750-registry-certificates\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.807328 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.807289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmrw\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-kube-api-access-xhmrw\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.807328 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.807308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f70e391d-9436-4989-b9ed-992782a0b750-image-registry-private-configuration\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.907998 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.907940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-bound-sa-token\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.907998 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.907975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f70e391d-9436-4989-b9ed-992782a0b750-trusted-ca\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.908159 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.908006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f70e391d-9436-4989-b9ed-992782a0b750-installation-pull-secrets\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.908159 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.908141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-registry-tls\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.908258 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.908184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f70e391d-9436-4989-b9ed-992782a0b750-registry-certificates\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.908330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.908311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmrw\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-kube-api-access-xhmrw\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.908433 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.908344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f70e391d-9436-4989-b9ed-992782a0b750-image-registry-private-configuration\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.908517 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.908431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f70e391d-9436-4989-b9ed-992782a0b750-ca-trust-extracted\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.908792 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.908767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f70e391d-9436-4989-b9ed-992782a0b750-ca-trust-extracted\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.909083 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.909057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f70e391d-9436-4989-b9ed-992782a0b750-registry-certificates\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.909205 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.909186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f70e391d-9436-4989-b9ed-992782a0b750-trusted-ca\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.910521 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.910501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f70e391d-9436-4989-b9ed-992782a0b750-installation-pull-secrets\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.910601 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.910532    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-registry-tls\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.910693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.910677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f70e391d-9436-4989-b9ed-992782a0b750-image-registry-private-configuration\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.915255 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.915232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-bound-sa-token\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.915635 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.915619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmrw\" (UniqueName: \"kubernetes.io/projected/f70e391d-9436-4989-b9ed-992782a0b750-kube-api-access-xhmrw\") pod \"image-registry-86d879bdf9-p9lqk\" (UID: \"f70e391d-9436-4989-b9ed-992782a0b750\") " pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.923568 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.923552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:01.992119 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:01.992089    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"]
Sep 11 03:51:01.995664 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:51:01.995638    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff00cced_0f7d_437b_a2f1_ccb730276edb.slice/crio-5a2a38a304857877efa3c9cf831d79b8ebfb62d59899bbc4bd23835324ce2777 WatchSource:0}: Error finding container 5a2a38a304857877efa3c9cf831d79b8ebfb62d59899bbc4bd23835324ce2777: Status 404 returned error can't find the container with id 5a2a38a304857877efa3c9cf831d79b8ebfb62d59899bbc4bd23835324ce2777
Sep 11 03:51:02.016567 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:02.016550    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp"]
Sep 11 03:51:02.018653 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:51:02.018630    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d8f8531_c5fa_43e7_a72b_552d78ffde7c.slice/crio-4536e1870c76966a1dff6c293ab0915d08c1ef0b8326612185a293c9ac9ff59d WatchSource:0}: Error finding container 4536e1870c76966a1dff6c293ab0915d08c1ef0b8326612185a293c9ac9ff59d: Status 404 returned error can't find the container with id 4536e1870c76966a1dff6c293ab0915d08c1ef0b8326612185a293c9ac9ff59d
Sep 11 03:51:02.033230 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:02.033201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl" event={"ID":"ff00cced-0f7d-437b-a2f1-ccb730276edb","Type":"ContainerStarted","Data":"5a2a38a304857877efa3c9cf831d79b8ebfb62d59899bbc4bd23835324ce2777"}
Sep 11 03:51:02.037432 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:02.034625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp" event={"ID":"7d8f8531-c5fa-43e7-a72b-552d78ffde7c","Type":"ContainerStarted","Data":"4536e1870c76966a1dff6c293ab0915d08c1ef0b8326612185a293c9ac9ff59d"}
Sep 11 03:51:02.189873 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:02.189853    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86d879bdf9-p9lqk"]
Sep 11 03:51:02.191865 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:51:02.191842    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70e391d_9436_4989_b9ed_992782a0b750.slice/crio-8facd98a42db5596faf78851ee22a8e6a3cf72b800b9d7de5ed0b3c5fc95e275 WatchSource:0}: Error finding container 8facd98a42db5596faf78851ee22a8e6a3cf72b800b9d7de5ed0b3c5fc95e275: Status 404 returned error can't find the container with id 8facd98a42db5596faf78851ee22a8e6a3cf72b800b9d7de5ed0b3c5fc95e275
Sep 11 03:51:02.933936 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:02.933905    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkpvh"
Sep 11 03:51:03.043436 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:03.042708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk" event={"ID":"f70e391d-9436-4989-b9ed-992782a0b750","Type":"ContainerStarted","Data":"22c60873b2f7e3cc22f5ca781e0f25642c9a29a5e65a3d7ad71e99dc4ce71232"}
Sep 11 03:51:03.043436 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:03.042765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk" event={"ID":"f70e391d-9436-4989-b9ed-992782a0b750","Type":"ContainerStarted","Data":"8facd98a42db5596faf78851ee22a8e6a3cf72b800b9d7de5ed0b3c5fc95e275"}
Sep 11 03:51:03.043872 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:03.043779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:03.066697 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:03.064154    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk" podStartSLOduration=2.064120991 podStartE2EDuration="2.064120991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:03.063315726 +0000 UTC m=+60.929335491" watchObservedRunningTime="2026-09-11 03:51:03.064120991 +0000 UTC m=+60.930140729"
Sep 11 03:51:05.017512 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:05.017480    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2trdb"
Sep 11 03:51:07.454685 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.454648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:51:07.455120 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.454728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:51:07.457204 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.457180    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 03:51:07.457903 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.457887    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 03:51:07.467963 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.467939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37437296-da5a-4a64-8dca-a89fa0263d79-original-pull-secret\") pod \"global-pull-secret-syncer-frxxs\" (UID: \"37437296-da5a-4a64-8dca-a89fa0263d79\") " pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:51:07.467963 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.467955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6f66daa-4ff1-45b4-93d6-cc8de58802b3-metrics-certs\") pod \"network-metrics-daemon-k57hw\" (UID: \"c6f66daa-4ff1-45b4-93d6-cc8de58802b3\") " pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:51:07.517151 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.517123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frxxs"
Sep 11 03:51:07.555671 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.555641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:51:07.558201 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.558181    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 03:51:07.568272 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.568244    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 03:51:07.580071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.580051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls7w4\" (UniqueName: \"kubernetes.io/projected/544a104a-4499-4f6e-a751-77db1ae1aca2-kube-api-access-ls7w4\") pod \"network-check-target-t7pq9\" (UID: \"544a104a-4499-4f6e-a751-77db1ae1aca2\") " pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:51:07.620209 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.620160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-58qzc\""
Sep 11 03:51:07.625073 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.625053    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-snbvm\""
Sep 11 03:51:07.628737 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.628715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k57hw"
Sep 11 03:51:07.634044 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.634029    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:51:07.988244 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.988212    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7pq9"]
Sep 11 03:51:07.993238 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.990757    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-frxxs"]
Sep 11 03:51:07.993238 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:51:07.992627    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37437296_da5a_4a64_8dca_a89fa0263d79.slice/crio-a76b75ab8b8bc58f638ab00f74b465ef41225afd7ef26293c613fb2c0e63da46 WatchSource:0}: Error finding container a76b75ab8b8bc58f638ab00f74b465ef41225afd7ef26293c613fb2c0e63da46: Status 404 returned error can't find the container with id a76b75ab8b8bc58f638ab00f74b465ef41225afd7ef26293c613fb2c0e63da46
Sep 11 03:51:07.995569 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:51:07.995546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544a104a_4499_4f6e_a751_77db1ae1aca2.slice/crio-6b4321a293ad2b5683df4ecdf9acb10513241eea9fc6211f3cefd0eb815bc9e7 WatchSource:0}: Error finding container 6b4321a293ad2b5683df4ecdf9acb10513241eea9fc6211f3cefd0eb815bc9e7: Status 404 returned error can't find the container with id 6b4321a293ad2b5683df4ecdf9acb10513241eea9fc6211f3cefd0eb815bc9e7
Sep 11 03:51:07.995653 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:07.995634    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k57hw"]
Sep 11 03:51:07.998498 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:51:07.998476    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f66daa_4ff1_45b4_93d6_cc8de58802b3.slice/crio-8ff2b7871a32e7689dc839225a579a131d72d80e8ba6f6bcb2d23d9c8fe78add WatchSource:0}: Error finding container 8ff2b7871a32e7689dc839225a579a131d72d80e8ba6f6bcb2d23d9c8fe78add: Status 404 returned error can't find the container with id 8ff2b7871a32e7689dc839225a579a131d72d80e8ba6f6bcb2d23d9c8fe78add
Sep 11 03:51:08.064300 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.064262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl" event={"ID":"ff00cced-0f7d-437b-a2f1-ccb730276edb","Type":"ContainerStarted","Data":"b40963e607f8e99bd21d15b22a5c5b37f655ed26b29a8f804f639bf9a7117a38"}
Sep 11 03:51:08.064423 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.064380    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:08.066094 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.066069    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl"
Sep 11 03:51:08.066462 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.066439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp" event={"ID":"7d8f8531-c5fa-43e7-a72b-552d78ffde7c","Type":"ContainerStarted","Data":"46e67c45d3bf79ba1ae2e577681b65a25df482063d09725ab29088c7d19c1ede"}
Sep 11 03:51:08.066551 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.066472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp" event={"ID":"7d8f8531-c5fa-43e7-a72b-552d78ffde7c","Type":"ContainerStarted","Data":"05f1deba6f65e1a32b9899ad76f10e2689c0879656bf941eb29c8fed2c8ace07"}
Sep 11 03:51:08.066551 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.066485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp" event={"ID":"7d8f8531-c5fa-43e7-a72b-552d78ffde7c","Type":"ContainerStarted","Data":"5dac6708611fec96a9dc02aaedef48c4695227b7d83962385bde328f6b68d175"}
Sep 11 03:51:08.067571 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.067548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k57hw" event={"ID":"c6f66daa-4ff1-45b4-93d6-cc8de58802b3","Type":"ContainerStarted","Data":"8ff2b7871a32e7689dc839225a579a131d72d80e8ba6f6bcb2d23d9c8fe78add"}
Sep 11 03:51:08.068600 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.068583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7pq9" event={"ID":"544a104a-4499-4f6e-a751-77db1ae1aca2","Type":"ContainerStarted","Data":"6b4321a293ad2b5683df4ecdf9acb10513241eea9fc6211f3cefd0eb815bc9e7"}
Sep 11 03:51:08.069583 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.069566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-frxxs" event={"ID":"37437296-da5a-4a64-8dca-a89fa0263d79","Type":"ContainerStarted","Data":"a76b75ab8b8bc58f638ab00f74b465ef41225afd7ef26293c613fb2c0e63da46"}
Sep 11 03:51:08.109774 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.109733    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669db9d7c6-7sntl" podStartSLOduration=1.383065402 podStartE2EDuration="7.109721835s" totalImagesPullingTime="5.726656433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:08.108881515 +0000 UTC m=+65.974901250" watchObservedRunningTime="2026-09-11 03:51:08.109721835 +0000 UTC m=+65.975741571"
Sep 11 03:51:08.147659 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:08.147601    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6f9966b6-bplrp" podStartSLOduration=1.431166262 podStartE2EDuration="7.147588553s" totalImagesPullingTime="5.716422291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:08.146311419 +0000 UTC m=+66.012331155" watchObservedRunningTime="2026-09-11 03:51:08.147588553 +0000 UTC m=+66.013608331"
Sep 11 03:51:10.080772 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.080734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k57hw" event={"ID":"c6f66daa-4ff1-45b4-93d6-cc8de58802b3","Type":"ContainerStarted","Data":"3a20c7535d58752b76c329cd0246b5070189ed0a1bd7a9228f027e786ecb1c1c"}
Sep 11 03:51:10.081220 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.080817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k57hw" event={"ID":"c6f66daa-4ff1-45b4-93d6-cc8de58802b3","Type":"ContainerStarted","Data":"280d48f8c33d96fea27750a9018f41546d74a60dad918004b05d612d5fb58593"}
Sep 11 03:51:10.097280 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.097218    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k57hw" podStartSLOduration=65.692912201 podStartE2EDuration="1m7.097200982s" totalImagesPullingTime="1.404288781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:10.095834103 +0000 UTC m=+67.961853862" watchObservedRunningTime="2026-09-11 03:51:10.097200982 +0000 UTC m=+67.963220719"
Sep 11 03:51:10.789961 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.789891    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wqk7x"]
Sep 11 03:51:10.795055 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.795025    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.797590 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.797565    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 03:51:10.797731 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.797623    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpvnw\""
Sep 11 03:51:10.797731 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.797570    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 03:51:10.798615 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.798548    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 03:51:10.798745 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.798617    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 03:51:10.805518 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.805454    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqk7x"]
Sep 11 03:51:10.881921 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.881892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7015feea-77c2-4835-a93b-25fce799b674-crio-socket\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.881921 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.881923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7015feea-77c2-4835-a93b-25fce799b674-data-volume\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.882109 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.881961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.882109 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.882026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.882109 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.882068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468sl\" (UniqueName: \"kubernetes.io/projected/7015feea-77c2-4835-a93b-25fce799b674-kube-api-access-468sl\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.982921 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.982889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.983087 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.982939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.983087 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.982978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468sl\" (UniqueName: \"kubernetes.io/projected/7015feea-77c2-4835-a93b-25fce799b674-kube-api-access-468sl\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.983087 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.983038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7015feea-77c2-4835-a93b-25fce799b674-crio-socket\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.983087 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:10.983038    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:10.983087 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.983062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7015feea-77c2-4835-a93b-25fce799b674-data-volume\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.983327 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:10.983126    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls podName:7015feea-77c2-4835-a93b-25fce799b674 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:11.483104895 +0000 UTC m=+69.349124628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqk7x" (UID: "7015feea-77c2-4835-a93b-25fce799b674") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:10.983327 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.983125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7015feea-77c2-4835-a93b-25fce799b674-crio-socket\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.983327 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:10.983255    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:51:10.983502 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:10.983334    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm podName:7015feea-77c2-4835-a93b-25fce799b674 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:11.483315041 +0000 UTC m=+69.349334765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqk7x" (UID: "7015feea-77c2-4835-a93b-25fce799b674") : configmap "kube-rbac-proxy" not found
Sep 11 03:51:10.983502 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.983398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7015feea-77c2-4835-a93b-25fce799b674-data-volume\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:10.992220 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:10.992197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-468sl\" (UniqueName: \"kubernetes.io/projected/7015feea-77c2-4835-a93b-25fce799b674-kube-api-access-468sl\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:11.487349 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:11.487313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:11.487794 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:11.487435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:11.487794 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:11.487467    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:51:11.487794 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:11.487548    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm podName:7015feea-77c2-4835-a93b-25fce799b674 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:12.487526515 +0000 UTC m=+70.353546246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqk7x" (UID: "7015feea-77c2-4835-a93b-25fce799b674") : configmap "kube-rbac-proxy" not found
Sep 11 03:51:11.487794 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:11.487555    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:11.487794 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:11.487608    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls podName:7015feea-77c2-4835-a93b-25fce799b674 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:12.487591436 +0000 UTC m=+70.353611151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqk7x" (UID: "7015feea-77c2-4835-a93b-25fce799b674") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:11.512390 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:11.512350    2566 patch_prober.go:28] interesting pod/image-registry-79fc457b8f-7nqkf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 03:51:11.512534 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:11.512428    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf" podUID="65a48c04-6e08-42ab-ad40-11acc7435662" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 03:51:12.496111 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:12.496075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:12.496666 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:12.496129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:12.496666 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:12.496244    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:51:12.496666 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:12.496254    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:12.496666 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:12.496312    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm podName:7015feea-77c2-4835-a93b-25fce799b674 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:14.496291764 +0000 UTC m=+72.362311495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqk7x" (UID: "7015feea-77c2-4835-a93b-25fce799b674") : configmap "kube-rbac-proxy" not found
Sep 11 03:51:12.496666 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:12.496332    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls podName:7015feea-77c2-4835-a93b-25fce799b674 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:14.496322213 +0000 UTC m=+72.362341928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqk7x" (UID: "7015feea-77c2-4835-a93b-25fce799b674") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:13.093993 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:13.093911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7pq9" event={"ID":"544a104a-4499-4f6e-a751-77db1ae1aca2","Type":"ContainerStarted","Data":"02617d1f38f92d89d2c122c1b178d8b62a42803a6b24d02dc9abe8b04540bda1"}
Sep 11 03:51:13.094151 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:13.094018    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:51:13.095492 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:13.095467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-frxxs" event={"ID":"37437296-da5a-4a64-8dca-a89fa0263d79","Type":"ContainerStarted","Data":"6c3dcc93985eda82c99959a1aa4333acd7884fd3e7b39959fe272cbb9abee277"}
Sep 11 03:51:13.123510 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:13.123465    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t7pq9" podStartSLOduration=65.346731297 podStartE2EDuration="1m10.123449588s" totalImagesPullingTime="4.776718291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:13.122171363 +0000 UTC m=+70.988191101" watchObservedRunningTime="2026-09-11 03:51:13.123449588 +0000 UTC m=+70.989469323"
Sep 11 03:51:14.192733 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:14.192706    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2trdb_6e3f4a25-3d98-48b6-9026-045c9e3df008/dns/0.log"
Sep 11 03:51:14.364858 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:14.364832    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2trdb_6e3f4a25-3d98-48b6-9026-045c9e3df008/kube-rbac-proxy/0.log"
Sep 11 03:51:14.510529 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:14.510503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:14.510638 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:14.510569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:14.510638 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:14.510616    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 03:51:14.510707 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:14.510666    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:14.510707 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:14.510675    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm podName:7015feea-77c2-4835-a93b-25fce799b674 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:18.510658244 +0000 UTC m=+76.376677964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqk7x" (UID: "7015feea-77c2-4835-a93b-25fce799b674") : configmap "kube-rbac-proxy" not found
Sep 11 03:51:14.510775 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:14.510710    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls podName:7015feea-77c2-4835-a93b-25fce799b674 nodeName:}" failed. No retries permitted until 2026-09-11 03:51:18.510699752 +0000 UTC m=+76.376719466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqk7x" (UID: "7015feea-77c2-4835-a93b-25fce799b674") : secret "insights-runtime-extractor-tls" not found
Sep 11 03:51:15.364106 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:15.364077    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2d7rk_82a364ce-5981-412b-9cf0-65da78af29d9/dns-node-resolver/0.log"
Sep 11 03:51:16.165017 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:16.164987    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79fc457b8f-7nqkf_65a48c04-6e08-42ab-ad40-11acc7435662/registry/0.log"
Sep 11 03:51:16.364376 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:16.364347    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86d879bdf9-p9lqk_f70e391d-9436-4989-b9ed-992782a0b750/registry/0.log"
Sep 11 03:51:16.763886 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:16.763864    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6lvvf_cdc800d9-e949-4c5e-9804-1f74c6b507b9/node-ca/0.log"
Sep 11 03:51:17.564883 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:17.564857    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-btpdt_4eeec831-824a-4089-87ff-a203bf1d9a87/serve-healthcheck-canary/0.log"
Sep 11 03:51:18.542111 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.542078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:18.542306 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.542128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:18.542807 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.542784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7015feea-77c2-4835-a93b-25fce799b674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:18.544804 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.544774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7015feea-77c2-4835-a93b-25fce799b674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqk7x\" (UID: \"7015feea-77c2-4835-a93b-25fce799b674\") " pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:18.606422 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.606382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqk7x"
Sep 11 03:51:18.620641 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.619674    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-frxxs" podStartSLOduration=70.834474607 podStartE2EDuration="1m15.619656288s" totalImagesPullingTime="4.785181681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:13.140044381 +0000 UTC m=+71.006064119" watchObservedRunningTime="2026-09-11 03:51:18.619656288 +0000 UTC m=+76.485676017"
Sep 11 03:51:18.621058 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.621037    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qcczh"]
Sep 11 03:51:18.623379 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.623359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.630478 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.630217    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 03:51:18.630930 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.630911    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 03:51:18.631149 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.631131    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 03:51:18.631313 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.631299    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 03:51:18.631493 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.631474    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 03:51:18.631888 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.631676    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 03:51:18.638092 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.638072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k2sm5\""
Sep 11 03:51:18.743453 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.743595 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-sys\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.743595 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-tls\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.743708 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-root\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.743763 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.743763 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-metrics-client-ca\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.743866 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbb2\" (UniqueName: \"kubernetes.io/projected/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-kube-api-access-frbb2\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.743866 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-textfile\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.743938 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.743878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-wtmp\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.844794 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-sys\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.844794 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-tls\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.844794 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-root\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845024 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845024 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-metrics-client-ca\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845024 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-sys\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845024 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-root\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845024 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbb2\" (UniqueName: \"kubernetes.io/projected/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-kube-api-access-frbb2\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845024 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.844973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-textfile\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845024 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.845000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-wtmp\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845456 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.845031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845456 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.845235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-wtmp\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845456 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.845388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-metrics-client-ca\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845666 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.845641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.845768 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.845713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-textfile\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.847122 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.847101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.847522 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.847504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-node-exporter-tls\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.852863 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.852841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbb2\" (UniqueName: \"kubernetes.io/projected/7577d019-76b8-4fdc-bfc2-c57d03cad2c6-kube-api-access-frbb2\") pod \"node-exporter-qcczh\" (UID: \"7577d019-76b8-4fdc-bfc2-c57d03cad2c6\") " pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.883525 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.883505    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqk7x"]
Sep 11 03:51:18.885911 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:51:18.885890    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7015feea_77c2_4835_a93b_25fce799b674.slice/crio-d3a44903161e50975f47cc63837e87f3be642459c4d5e521c52b17be22650236 WatchSource:0}: Error finding container d3a44903161e50975f47cc63837e87f3be642459c4d5e521c52b17be22650236: Status 404 returned error can't find the container with id d3a44903161e50975f47cc63837e87f3be642459c4d5e521c52b17be22650236
Sep 11 03:51:18.934168 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:18.934149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qcczh"
Sep 11 03:51:18.942929 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:51:18.942907    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7577d019_76b8_4fdc_bfc2_c57d03cad2c6.slice/crio-8cefcfbc53f3ca395340ef1d331917a12d1af7926abf29ea94c798e4cb819a67 WatchSource:0}: Error finding container 8cefcfbc53f3ca395340ef1d331917a12d1af7926abf29ea94c798e4cb819a67: Status 404 returned error can't find the container with id 8cefcfbc53f3ca395340ef1d331917a12d1af7926abf29ea94c798e4cb819a67
Sep 11 03:51:19.122071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:19.121994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcczh" event={"ID":"7577d019-76b8-4fdc-bfc2-c57d03cad2c6","Type":"ContainerStarted","Data":"8cefcfbc53f3ca395340ef1d331917a12d1af7926abf29ea94c798e4cb819a67"}
Sep 11 03:51:19.123432 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:19.123397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqk7x" event={"ID":"7015feea-77c2-4835-a93b-25fce799b674","Type":"ContainerStarted","Data":"938ff82e35fdfd7da0a3e06e13e1defb86c24e9670c9f272c7a658e768a0280b"}
Sep 11 03:51:19.123540 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:19.123451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqk7x" event={"ID":"7015feea-77c2-4835-a93b-25fce799b674","Type":"ContainerStarted","Data":"d3a44903161e50975f47cc63837e87f3be642459c4d5e521c52b17be22650236"}
Sep 11 03:51:20.128848 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:20.128808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcczh" event={"ID":"7577d019-76b8-4fdc-bfc2-c57d03cad2c6","Type":"ContainerStarted","Data":"aaa8f2fc462449bbf96d053a00877981b9e5063dccabbf76f97a745aaa28bda1"}
Sep 11 03:51:20.130796 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:20.130771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqk7x" event={"ID":"7015feea-77c2-4835-a93b-25fce799b674","Type":"ContainerStarted","Data":"bbf98d96c535e369bfbfe30da763cace3b6392f23a7348bb529974ea51733b93"}
Sep 11 03:51:21.136550 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:21.136510    2566 generic.go:356] "Generic (PLEG): container finished" podID="7577d019-76b8-4fdc-bfc2-c57d03cad2c6" containerID="aaa8f2fc462449bbf96d053a00877981b9e5063dccabbf76f97a745aaa28bda1" exitCode=0
Sep 11 03:51:21.136954 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:21.136567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcczh" event={"ID":"7577d019-76b8-4fdc-bfc2-c57d03cad2c6","Type":"ContainerDied","Data":"aaa8f2fc462449bbf96d053a00877981b9e5063dccabbf76f97a745aaa28bda1"}
Sep 11 03:51:21.512726 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:21.512699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:51:22.142916 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:22.142874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcczh" event={"ID":"7577d019-76b8-4fdc-bfc2-c57d03cad2c6","Type":"ContainerStarted","Data":"14dddc3c0f52bc5260bee58e5572597d08b1d25cede8da82068ff4b76e3d67dc"}
Sep 11 03:51:22.143340 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:22.142938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcczh" event={"ID":"7577d019-76b8-4fdc-bfc2-c57d03cad2c6","Type":"ContainerStarted","Data":"61fa4853fb36cf8baa5ea58fe50c567ccf1f32a20c313e745c10da79b1a90d9e"}
Sep 11 03:51:22.145305 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:22.145267    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqk7x" event={"ID":"7015feea-77c2-4835-a93b-25fce799b674","Type":"ContainerStarted","Data":"82373f120c0733e7e933ae0742f6a4cbf3dbd6886e481a7dc03257f7ea3352d5"}
Sep 11 03:51:22.160744 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:22.160688    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qcczh" podStartSLOduration=3.042963422 podStartE2EDuration="4.16067s" totalImagesPullingTime="1.056146125s" totalInitContainerRuntime="61.560453ms" isStatefulPod=true podCreationTimestamp="2026-09-11 03:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:22.16032277 +0000 UTC m=+80.026342507" watchObservedRunningTime="2026-09-11 03:51:22.16067 +0000 UTC m=+80.026689735"
Sep 11 03:51:22.181339 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:22.181298    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wqk7x" podStartSLOduration=9.872758166 podStartE2EDuration="12.181284836s" totalImagesPullingTime="2.30852667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:51:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:51:22.181264966 +0000 UTC m=+80.047284715" watchObservedRunningTime="2026-09-11 03:51:22.181284836 +0000 UTC m=+80.047304572"
Sep 11 03:51:25.056210 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:25.056094    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86d879bdf9-p9lqk"
Sep 11 03:51:26.523502 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.523437    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf" podUID="65a48c04-6e08-42ab-ad40-11acc7435662" containerName="registry" containerID="cri-o://bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140" gracePeriod=30
Sep 11 03:51:26.764287 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.764264    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:51:26.805614 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.805549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-image-registry-private-configuration\") pod \"65a48c04-6e08-42ab-ad40-11acc7435662\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") "
Sep 11 03:51:26.805614 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.805581    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") pod \"65a48c04-6e08-42ab-ad40-11acc7435662\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") "
Sep 11 03:51:26.805758 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.805601    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65a48c04-6e08-42ab-ad40-11acc7435662-ca-trust-extracted\") pod \"65a48c04-6e08-42ab-ad40-11acc7435662\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") "
Sep 11 03:51:26.805758 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.805728    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-registry-certificates\") pod \"65a48c04-6e08-42ab-ad40-11acc7435662\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") "
Sep 11 03:51:26.805865 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.805770    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-installation-pull-secrets\") pod \"65a48c04-6e08-42ab-ad40-11acc7435662\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") "
Sep 11 03:51:26.805865 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.805790    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5bgw\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-kube-api-access-z5bgw\") pod \"65a48c04-6e08-42ab-ad40-11acc7435662\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") "
Sep 11 03:51:26.805865 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.805820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-bound-sa-token\") pod \"65a48c04-6e08-42ab-ad40-11acc7435662\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") "
Sep 11 03:51:26.805865 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.805851    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-trusted-ca\") pod \"65a48c04-6e08-42ab-ad40-11acc7435662\" (UID: \"65a48c04-6e08-42ab-ad40-11acc7435662\") "
Sep 11 03:51:26.806180 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.806120    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "65a48c04-6e08-42ab-ad40-11acc7435662" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:26.806614 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.806575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "65a48c04-6e08-42ab-ad40-11acc7435662" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:51:26.808362 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.808338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "65a48c04-6e08-42ab-ad40-11acc7435662" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:51:26.808540 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.808501    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "65a48c04-6e08-42ab-ad40-11acc7435662" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:51:26.808540 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.808508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-kube-api-access-z5bgw" (OuterVolumeSpecName: "kube-api-access-z5bgw") pod "65a48c04-6e08-42ab-ad40-11acc7435662" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662"). InnerVolumeSpecName "kube-api-access-z5bgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:51:26.808674 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.808654    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "65a48c04-6e08-42ab-ad40-11acc7435662" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:51:26.809083 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.809063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "65a48c04-6e08-42ab-ad40-11acc7435662" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:51:26.815044 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.815021    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a48c04-6e08-42ab-ad40-11acc7435662-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "65a48c04-6e08-42ab-ad40-11acc7435662" (UID: "65a48c04-6e08-42ab-ad40-11acc7435662"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:51:26.907217 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.907195    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-registry-certificates\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:51:26.907217 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.907217    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-installation-pull-secrets\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:51:26.907330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.907226    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5bgw\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-kube-api-access-z5bgw\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:51:26.907330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.907235    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-bound-sa-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:51:26.907330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.907244    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65a48c04-6e08-42ab-ad40-11acc7435662-trusted-ca\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:51:26.907330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.907253    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65a48c04-6e08-42ab-ad40-11acc7435662-image-registry-private-configuration\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:51:26.907330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.907261    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65a48c04-6e08-42ab-ad40-11acc7435662-registry-tls\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:51:26.907330 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:26.907269    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65a48c04-6e08-42ab-ad40-11acc7435662-ca-trust-extracted\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:51:27.162378 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.162323    2566 generic.go:356] "Generic (PLEG): container finished" podID="65a48c04-6e08-42ab-ad40-11acc7435662" containerID="bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140" exitCode=0
Sep 11 03:51:27.162378 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.162362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf" event={"ID":"65a48c04-6e08-42ab-ad40-11acc7435662","Type":"ContainerDied","Data":"bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140"}
Sep 11 03:51:27.162378 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.162367    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf"
Sep 11 03:51:27.162563 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.162394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fc457b8f-7nqkf" event={"ID":"65a48c04-6e08-42ab-ad40-11acc7435662","Type":"ContainerDied","Data":"e68b1c827367ab7c433403ed634aeb172d49a7ac71f99481d2e1ecc14ff52daa"}
Sep 11 03:51:27.162563 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.162432    2566 scope.go:117] "RemoveContainer" containerID="bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140"
Sep 11 03:51:27.171141 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.171107    2566 scope.go:117] "RemoveContainer" containerID="bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140"
Sep 11 03:51:27.171419 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:51:27.171389    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140\": container with ID starting with bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140 not found: ID does not exist" containerID="bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140"
Sep 11 03:51:27.171477 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.171431    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140"} err="failed to get container status \"bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140\": rpc error: code = NotFound desc = could not find container \"bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140\": container with ID starting with bf76aaf9c8832462de43d3bc9f9678d3bfb2e8696e69990fecb5e213a39db140 not found: ID does not exist"
Sep 11 03:51:27.181574 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.181555    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79fc457b8f-7nqkf"]
Sep 11 03:51:27.185427 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:27.185395    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79fc457b8f-7nqkf"]
Sep 11 03:51:28.610920 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:28.610887    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a48c04-6e08-42ab-ad40-11acc7435662" path="/var/lib/kubelet/pods/65a48c04-6e08-42ab-ad40-11acc7435662/volumes"
Sep 11 03:51:44.102236 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:51:44.102207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t7pq9"
Sep 11 03:54:51.412263 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.412227    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"]
Sep 11 03:54:51.412690 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.412477    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a48c04-6e08-42ab-ad40-11acc7435662" containerName="registry"
Sep 11 03:54:51.412690 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.412488    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a48c04-6e08-42ab-ad40-11acc7435662" containerName="registry"
Sep 11 03:54:51.412690 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.412538    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a48c04-6e08-42ab-ad40-11acc7435662" containerName="registry"
Sep 11 03:54:51.415156 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.415138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"
Sep 11 03:54:51.418173 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.418150    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 03:54:51.418251 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.418230    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-st6kb\""
Sep 11 03:54:51.419133 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.419117    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 03:54:51.434054 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.434028    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"]
Sep 11 03:54:51.511937 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.511915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436c563e-bce9-4235-8ba4-c1ef14eee54a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ftqtb\" (UID: \"436c563e-bce9-4235-8ba4-c1ef14eee54a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"
Sep 11 03:54:51.512056 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.511950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjw9\" (UniqueName: \"kubernetes.io/projected/436c563e-bce9-4235-8ba4-c1ef14eee54a-kube-api-access-txjw9\") pod \"openshift-lws-operator-fd8ccff4c-ftqtb\" (UID: \"436c563e-bce9-4235-8ba4-c1ef14eee54a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"
Sep 11 03:54:51.612977 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.612949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjw9\" (UniqueName: \"kubernetes.io/projected/436c563e-bce9-4235-8ba4-c1ef14eee54a-kube-api-access-txjw9\") pod \"openshift-lws-operator-fd8ccff4c-ftqtb\" (UID: \"436c563e-bce9-4235-8ba4-c1ef14eee54a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"
Sep 11 03:54:51.613071 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.613015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436c563e-bce9-4235-8ba4-c1ef14eee54a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ftqtb\" (UID: \"436c563e-bce9-4235-8ba4-c1ef14eee54a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"
Sep 11 03:54:51.613393 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.613376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436c563e-bce9-4235-8ba4-c1ef14eee54a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ftqtb\" (UID: \"436c563e-bce9-4235-8ba4-c1ef14eee54a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"
Sep 11 03:54:51.624224 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.624207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjw9\" (UniqueName: \"kubernetes.io/projected/436c563e-bce9-4235-8ba4-c1ef14eee54a-kube-api-access-txjw9\") pod \"openshift-lws-operator-fd8ccff4c-ftqtb\" (UID: \"436c563e-bce9-4235-8ba4-c1ef14eee54a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"
Sep 11 03:54:51.723750 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.723731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"
Sep 11 03:54:51.991525 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:51.991504    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb"]
Sep 11 03:54:51.993724 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:54:51.993694    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod436c563e_bce9_4235_8ba4_c1ef14eee54a.slice/crio-c518cc98fb31b29f1cdc504fa5e8be0df4a0737d7507efaee7de1ce9e4d35ee3 WatchSource:0}: Error finding container c518cc98fb31b29f1cdc504fa5e8be0df4a0737d7507efaee7de1ce9e4d35ee3: Status 404 returned error can't find the container with id c518cc98fb31b29f1cdc504fa5e8be0df4a0737d7507efaee7de1ce9e4d35ee3
Sep 11 03:54:52.819003 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:52.818963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb" event={"ID":"436c563e-bce9-4235-8ba4-c1ef14eee54a","Type":"ContainerStarted","Data":"c518cc98fb31b29f1cdc504fa5e8be0df4a0737d7507efaee7de1ce9e4d35ee3"}
Sep 11 03:54:54.828194 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:54.828111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb" event={"ID":"436c563e-bce9-4235-8ba4-c1ef14eee54a","Type":"ContainerStarted","Data":"f9c85ddfd8271a369d661f0dea435749cf9f3188466413e5cba5be7665dfa6c4"}
Sep 11 03:54:54.848075 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:54:54.848024    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ftqtb" podStartSLOduration=1.35184897 podStartE2EDuration="3.84800852s" totalImagesPullingTime="2.49615955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:54.846242357 +0000 UTC m=+292.712262092" watchObservedRunningTime="2026-09-11 03:54:54.84800852 +0000 UTC m=+292.714028255"
Sep 11 03:55:02.558919 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:02.558890    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 03:55:02.561962 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:02.561942    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 03:55:02.566373 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:02.566261    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 03:55:11.303904 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.303865    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"]
Sep 11 03:55:11.307217 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.307198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.309793 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.309773    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 03:55:11.309889 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.309808    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 03:55:11.309889 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.309773    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 03:55:11.309987 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.309941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cm9z7\""
Sep 11 03:55:11.310307 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.310291    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 03:55:11.326082 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.326061    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"]
Sep 11 03:55:11.450234 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.450210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.450347 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.450241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.450347 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.450264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwm4h\" (UniqueName: \"kubernetes.io/projected/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-kube-api-access-hwm4h\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.450486 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.450361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-manifests\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.450486 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.450450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-tmp\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.550675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.550650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwm4h\" (UniqueName: \"kubernetes.io/projected/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-kube-api-access-hwm4h\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.550785 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.550685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-manifests\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.550785 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.550712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-tmp\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.550785 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.550738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.550785 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.550781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.551119 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.551099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-manifests\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.553133 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.553113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-tmp\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.553351 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.553334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.553392 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.553370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.568402 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.568354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwm4h\" (UniqueName: \"kubernetes.io/projected/7dbdf4d6-5fcf-4ade-adc5-78f535593c2c-kube-api-access-hwm4h\") pod \"opendatahub-operator-controller-manager-97cf4878d-bp9vs\" (UID: \"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c\") " pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.617361 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.617329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:11.896642 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.896618    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"]
Sep 11 03:55:11.898207 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:55:11.898181    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbdf4d6_5fcf_4ade_adc5_78f535593c2c.slice/crio-d73e0729f250c5c69d927879e1735b3dada2e7b6c6fb033f7e60e6f3e36f1379 WatchSource:0}: Error finding container d73e0729f250c5c69d927879e1735b3dada2e7b6c6fb033f7e60e6f3e36f1379: Status 404 returned error can't find the container with id d73e0729f250c5c69d927879e1735b3dada2e7b6c6fb033f7e60e6f3e36f1379
Sep 11 03:55:11.900025 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:11.900006    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 03:55:12.891210 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:12.891164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs" event={"ID":"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c","Type":"ContainerStarted","Data":"d73e0729f250c5c69d927879e1735b3dada2e7b6c6fb033f7e60e6f3e36f1379"}
Sep 11 03:55:14.900676 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:14.900603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs" event={"ID":"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c","Type":"ContainerStarted","Data":"d10f8c51e887accd051f0c3b4370020f92def308b5fc2390c00c6287e83319d8"}
Sep 11 03:55:15.905105 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:15.905069    2566 generic.go:356] "Generic (PLEG): container finished" podID="7dbdf4d6-5fcf-4ade-adc5-78f535593c2c" containerID="d10f8c51e887accd051f0c3b4370020f92def308b5fc2390c00c6287e83319d8" exitCode=0
Sep 11 03:55:15.905441 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:15.905116    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs" event={"ID":"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c","Type":"ContainerDied","Data":"d10f8c51e887accd051f0c3b4370020f92def308b5fc2390c00c6287e83319d8"}
Sep 11 03:55:16.910311 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:16.910275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs" event={"ID":"7dbdf4d6-5fcf-4ade-adc5-78f535593c2c","Type":"ContainerStarted","Data":"45352e422cfe96aae5021b026d9f2ad6122a443161d97e0245f00ae21063d5c4"}
Sep 11 03:55:16.910675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:16.910398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:16.946433 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:16.946359    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs" podStartSLOduration=1.637352728 podStartE2EDuration="5.94633809s" totalImagesPullingTime="3.245222685s" totalInitContainerRuntime="1.063762677s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:16.943948097 +0000 UTC m=+314.809967834" watchObservedRunningTime="2026-09-11 03:55:16.94633809 +0000 UTC m=+314.812357827"
Sep 11 03:55:27.916523 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:27.916493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-97cf4878d-bp9vs"
Sep 11 03:55:32.985505 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.985475    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"]
Sep 11 03:55:32.992135 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.992114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:32.994779 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.994751    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 03:55:32.994779 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.994751    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-67rtg\""
Sep 11 03:55:32.995669 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.995649    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 03:55:32.995796 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.995758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 03:55:32.995889 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.995872    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 03:55:32.997323 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.997299    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"]
Sep 11 03:55:32.997830 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.997806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tls-certs\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:32.997946 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.997856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tmp\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:32.997946 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:32.997927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68b8n\" (UniqueName: \"kubernetes.io/projected/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-kube-api-access-68b8n\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:33.098430 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:33.098397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68b8n\" (UniqueName: \"kubernetes.io/projected/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-kube-api-access-68b8n\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:33.098525 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:33.098457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tls-certs\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:33.098578 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:33.098530    2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 03:55:33.098613 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:33.098581    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tls-certs podName:bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d nodeName:}" failed. No retries permitted until 2026-09-11 03:55:33.598564889 +0000 UTC m=+331.464584603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tls-certs") pod "kube-auth-proxy-5d5f7bb976-c5lcl" (UID: "bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d") : secret "kube-auth-proxy-tls" not found
Sep 11 03:55:33.098669 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:33.098627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tmp\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:33.100616 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:33.100595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tmp\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:33.107125 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:33.107101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68b8n\" (UniqueName: \"kubernetes.io/projected/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-kube-api-access-68b8n\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:33.602168 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:33.602134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tls-certs\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:33.604369 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:33.604342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d-tls-certs\") pod \"kube-auth-proxy-5d5f7bb976-c5lcl\" (UID: \"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d\") " pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:33.903305 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:33.903244    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"
Sep 11 03:55:34.181319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.181241    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl"]
Sep 11 03:55:34.184358 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:55:34.184321    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd285f9c_6e7a_4461_8ac7_dfdcb89cb22d.slice/crio-1deeeba58d62a05896a55df9bb18a1651eaede4940c405fb2dc1722bf76f84d2 WatchSource:0}: Error finding container 1deeeba58d62a05896a55df9bb18a1651eaede4940c405fb2dc1722bf76f84d2: Status 404 returned error can't find the container with id 1deeeba58d62a05896a55df9bb18a1651eaede4940c405fb2dc1722bf76f84d2
Sep 11 03:55:34.271379 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.271352    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"]
Sep 11 03:55:34.273922 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.273906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.276516 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.276465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 03:55:34.276516 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.276468    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 03:55:34.276793 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.276521    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 03:55:34.276872 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.276827    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-khghr\""
Sep 11 03:55:34.276918 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.276830    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 03:55:34.309298 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.309268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hld\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-kube-api-access-c5hld\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.309393 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.309312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.309479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.309402    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.309523 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.309493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.309565 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.309537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f845e7-50f6-4d23-8bfb-48ad246ee505-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.309610 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.309567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.309610 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.309590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.330254 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.330234    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"]
Sep 11 03:55:34.410354 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.410333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.410459 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.410362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.410459 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.410389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f845e7-50f6-4d23-8bfb-48ad246ee505-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.410459 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.410437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.410581 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.410463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.410581 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.410486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hld\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-kube-api-access-c5hld\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.410581 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.410510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.411126 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.411107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.413006 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.412982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.413114 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.413037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f845e7-50f6-4d23-8bfb-48ad246ee505-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.413114 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.413097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.413336 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.413319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.418500 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.418477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.418633 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.418612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hld\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-kube-api-access-c5hld\") pod \"istiod-openshift-gateway-7b7995d5cd-9cg2m\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.583512 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.583490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:34.858275 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.858158    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"]
Sep 11 03:55:34.862376 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:55:34.862349    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f845e7_50f6_4d23_8bfb_48ad246ee505.slice/crio-d74641b9e0436e51bcb795c97e95348d209560d528319bfe471fa929b197686a WatchSource:0}: Error finding container d74641b9e0436e51bcb795c97e95348d209560d528319bfe471fa929b197686a: Status 404 returned error can't find the container with id d74641b9e0436e51bcb795c97e95348d209560d528319bfe471fa929b197686a
Sep 11 03:55:34.975669 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.975622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl" event={"ID":"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d","Type":"ContainerStarted","Data":"1deeeba58d62a05896a55df9bb18a1651eaede4940c405fb2dc1722bf76f84d2"}
Sep 11 03:55:34.977198 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:34.977160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m" event={"ID":"88f845e7-50f6-4d23-8bfb-48ad246ee505","Type":"ContainerStarted","Data":"d74641b9e0436e51bcb795c97e95348d209560d528319bfe471fa929b197686a"}
Sep 11 03:55:36.990865 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:36.990814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl" event={"ID":"bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d","Type":"ContainerStarted","Data":"fe1c8910118ab4418cf2f2115409085a4bc05fded889eb7ba26a496913e96d45"}
Sep 11 03:55:37.009257 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:37.009206    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d5f7bb976-c5lcl" podStartSLOduration=2.54125211 podStartE2EDuration="5.009185993s" totalImagesPullingTime="2.467933883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:37.007142957 +0000 UTC m=+334.873162704" watchObservedRunningTime="2026-09-11 03:55:37.009185993 +0000 UTC m=+334.875205730"
Sep 11 03:55:38.435929 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:38.435895    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:55:38.436201 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:38.435978    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:55:39.000323 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:39.000291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m" event={"ID":"88f845e7-50f6-4d23-8bfb-48ad246ee505","Type":"ContainerStarted","Data":"871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865"}
Sep 11 03:55:39.000547 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:39.000526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:39.002131 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:39.002109    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9cg2m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 03:55:39.002199 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:39.002152    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m" podUID="88f845e7-50f6-4d23-8bfb-48ad246ee505" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 03:55:39.029937 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:39.029884    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m" podStartSLOduration=1.459121949 podStartE2EDuration="5.029864224s" totalImagesPullingTime="3.570742275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:39.027366315 +0000 UTC m=+336.893386054" watchObservedRunningTime="2026-09-11 03:55:39.029864224 +0000 UTC m=+336.895883969"
Sep 11 03:55:40.004963 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:40.004931    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9cg2m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 03:55:40.005319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:40.004990    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m" podUID="88f845e7-50f6-4d23-8bfb-48ad246ee505" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 03:55:41.677617 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.677541    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4kqzt"]
Sep 11 03:55:41.682470 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.682448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:41.685155 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.685135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 03:55:41.691254 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.691110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4kqzt"]
Sep 11 03:55:41.766537 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.766508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndn4t\" (UniqueName: \"kubernetes.io/projected/46b23d79-7b2e-4781-aadb-8abb5b8e224c-kube-api-access-ndn4t\") pod \"model-serving-api-54d89bd5fc-4kqzt\" (UID: \"46b23d79-7b2e-4781-aadb-8abb5b8e224c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:41.766690 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.766543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs\") pod \"model-serving-api-54d89bd5fc-4kqzt\" (UID: \"46b23d79-7b2e-4781-aadb-8abb5b8e224c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:41.867924 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.867887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndn4t\" (UniqueName: \"kubernetes.io/projected/46b23d79-7b2e-4781-aadb-8abb5b8e224c-kube-api-access-ndn4t\") pod \"model-serving-api-54d89bd5fc-4kqzt\" (UID: \"46b23d79-7b2e-4781-aadb-8abb5b8e224c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:41.868086 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.867927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs\") pod \"model-serving-api-54d89bd5fc-4kqzt\" (UID: \"46b23d79-7b2e-4781-aadb-8abb5b8e224c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:41.868086 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:41.868042    2566 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 11 03:55:41.868172 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:41.868107    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs podName:46b23d79-7b2e-4781-aadb-8abb5b8e224c nodeName:}" failed. No retries permitted until 2026-09-11 03:55:42.368087863 +0000 UTC m=+340.234107582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs") pod "model-serving-api-54d89bd5fc-4kqzt" (UID: "46b23d79-7b2e-4781-aadb-8abb5b8e224c") : secret "model-serving-api-tls" not found
Sep 11 03:55:41.875865 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:41.875841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndn4t\" (UniqueName: \"kubernetes.io/projected/46b23d79-7b2e-4781-aadb-8abb5b8e224c-kube-api-access-ndn4t\") pod \"model-serving-api-54d89bd5fc-4kqzt\" (UID: \"46b23d79-7b2e-4781-aadb-8abb5b8e224c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:42.371721 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:42.371684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs\") pod \"model-serving-api-54d89bd5fc-4kqzt\" (UID: \"46b23d79-7b2e-4781-aadb-8abb5b8e224c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:42.371926 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:42.371814    2566 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 11 03:55:42.371926 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:42.371867    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs podName:46b23d79-7b2e-4781-aadb-8abb5b8e224c nodeName:}" failed. No retries permitted until 2026-09-11 03:55:43.371853138 +0000 UTC m=+341.237872851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs") pod "model-serving-api-54d89bd5fc-4kqzt" (UID: "46b23d79-7b2e-4781-aadb-8abb5b8e224c") : secret "model-serving-api-tls" not found
Sep 11 03:55:43.005879 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:43.005852    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:43.379803 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:43.379703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs\") pod \"model-serving-api-54d89bd5fc-4kqzt\" (UID: \"46b23d79-7b2e-4781-aadb-8abb5b8e224c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:43.382082 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:43.382053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b23d79-7b2e-4781-aadb-8abb5b8e224c-tls-certs\") pod \"model-serving-api-54d89bd5fc-4kqzt\" (UID: \"46b23d79-7b2e-4781-aadb-8abb5b8e224c\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:43.495945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:43.495906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:43.773591 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:43.773559    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4kqzt"]
Sep 11 03:55:43.777167 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:55:43.777137    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b23d79_7b2e_4781_aadb_8abb5b8e224c.slice/crio-d6f5cd8cf883a50763200d9ca1fe4d5d1be3ea1479c98430f93af99c0672b273 WatchSource:0}: Error finding container d6f5cd8cf883a50763200d9ca1fe4d5d1be3ea1479c98430f93af99c0672b273: Status 404 returned error can't find the container with id d6f5cd8cf883a50763200d9ca1fe4d5d1be3ea1479c98430f93af99c0672b273
Sep 11 03:55:44.022185 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:44.022153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt" event={"ID":"46b23d79-7b2e-4781-aadb-8abb5b8e224c","Type":"ContainerStarted","Data":"d6f5cd8cf883a50763200d9ca1fe4d5d1be3ea1479c98430f93af99c0672b273"}
Sep 11 03:55:46.033109 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:46.033076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt" event={"ID":"46b23d79-7b2e-4781-aadb-8abb5b8e224c","Type":"ContainerStarted","Data":"3df22c36bb9ce0ee00ca7fe73de6fd09e673d3f607287bf5542067e93efe3f0b"}
Sep 11 03:55:46.033524 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:46.033206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:55:46.050080 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:46.050033    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt" podStartSLOduration=3.69924864 podStartE2EDuration="5.050017998s" totalImagesPullingTime="1.350769358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:46.048869519 +0000 UTC m=+343.914889254" watchObservedRunningTime="2026-09-11 03:55:46.050017998 +0000 UTC m=+343.916037769"
Sep 11 03:55:50.895112 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.895081    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"]
Sep 11 03:55:50.898475 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.898455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:50.904346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.901356    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hhh6x\""
Sep 11 03:55:50.904346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.901481    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 03:55:50.904346 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.903379    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"]
Sep 11 03:55:50.910476 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.908439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:50.911905 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.911851    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 03:55:50.912019 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.911943    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rbrdx\""
Sep 11 03:55:50.923731 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.923707    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"]
Sep 11 03:55:50.935063 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.935041    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"]
Sep 11 03:55:50.942225 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.942202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhd8f\" (UniqueName: \"kubernetes.io/projected/99aabf96-b1fe-4c82-af00-e9e034e69e14-kube-api-access-zhd8f\") pod \"llmisvc-controller-manager-777c6bd7d5-t6kbc\" (UID: \"99aabf96-b1fe-4c82-af00-e9e034e69e14\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:50.942325 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.942238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16b4f397-7fdf-4223-9c6a-3100c148db24-cert\") pod \"kserve-controller-manager-d6f4b65b8-9gt2l\" (UID: \"16b4f397-7fdf-4223-9c6a-3100c148db24\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:50.942325 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.942259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6kk\" (UniqueName: \"kubernetes.io/projected/16b4f397-7fdf-4223-9c6a-3100c148db24-kube-api-access-nz6kk\") pod \"kserve-controller-manager-d6f4b65b8-9gt2l\" (UID: \"16b4f397-7fdf-4223-9c6a-3100c148db24\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:50.942325 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:50.942284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99aabf96-b1fe-4c82-af00-e9e034e69e14-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t6kbc\" (UID: \"99aabf96-b1fe-4c82-af00-e9e034e69e14\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:51.043424 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.043380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhd8f\" (UniqueName: \"kubernetes.io/projected/99aabf96-b1fe-4c82-af00-e9e034e69e14-kube-api-access-zhd8f\") pod \"llmisvc-controller-manager-777c6bd7d5-t6kbc\" (UID: \"99aabf96-b1fe-4c82-af00-e9e034e69e14\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:51.043590 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.043438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16b4f397-7fdf-4223-9c6a-3100c148db24-cert\") pod \"kserve-controller-manager-d6f4b65b8-9gt2l\" (UID: \"16b4f397-7fdf-4223-9c6a-3100c148db24\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:51.043590 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.043464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6kk\" (UniqueName: \"kubernetes.io/projected/16b4f397-7fdf-4223-9c6a-3100c148db24-kube-api-access-nz6kk\") pod \"kserve-controller-manager-d6f4b65b8-9gt2l\" (UID: \"16b4f397-7fdf-4223-9c6a-3100c148db24\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:51.043590 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.043499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99aabf96-b1fe-4c82-af00-e9e034e69e14-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t6kbc\" (UID: \"99aabf96-b1fe-4c82-af00-e9e034e69e14\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:51.043752 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:51.043638    2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 11 03:55:51.043752 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:51.043709    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99aabf96-b1fe-4c82-af00-e9e034e69e14-cert podName:99aabf96-b1fe-4c82-af00-e9e034e69e14 nodeName:}" failed. No retries permitted until 2026-09-11 03:55:51.543688151 +0000 UTC m=+349.409707866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99aabf96-b1fe-4c82-af00-e9e034e69e14-cert") pod "llmisvc-controller-manager-777c6bd7d5-t6kbc" (UID: "99aabf96-b1fe-4c82-af00-e9e034e69e14") : secret "llmisvc-webhook-server-cert" not found
Sep 11 03:55:51.045867 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.045839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16b4f397-7fdf-4223-9c6a-3100c148db24-cert\") pod \"kserve-controller-manager-d6f4b65b8-9gt2l\" (UID: \"16b4f397-7fdf-4223-9c6a-3100c148db24\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:51.055183 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.055157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6kk\" (UniqueName: \"kubernetes.io/projected/16b4f397-7fdf-4223-9c6a-3100c148db24-kube-api-access-nz6kk\") pod \"kserve-controller-manager-d6f4b65b8-9gt2l\" (UID: \"16b4f397-7fdf-4223-9c6a-3100c148db24\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:51.055440 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.055421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhd8f\" (UniqueName: \"kubernetes.io/projected/99aabf96-b1fe-4c82-af00-e9e034e69e14-kube-api-access-zhd8f\") pod \"llmisvc-controller-manager-777c6bd7d5-t6kbc\" (UID: \"99aabf96-b1fe-4c82-af00-e9e034e69e14\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:51.226110 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.226077    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:51.502662 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.502590    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"]
Sep 11 03:55:51.505242 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:55:51.505207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b4f397_7fdf_4223_9c6a_3100c148db24.slice/crio-64e0df7e1b83e1df0426d01717673638c463166686657199f30877a8c60bad57 WatchSource:0}: Error finding container 64e0df7e1b83e1df0426d01717673638c463166686657199f30877a8c60bad57: Status 404 returned error can't find the container with id 64e0df7e1b83e1df0426d01717673638c463166686657199f30877a8c60bad57
Sep 11 03:55:51.549230 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.549208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99aabf96-b1fe-4c82-af00-e9e034e69e14-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t6kbc\" (UID: \"99aabf96-b1fe-4c82-af00-e9e034e69e14\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:51.551826 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.551808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99aabf96-b1fe-4c82-af00-e9e034e69e14-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-t6kbc\" (UID: \"99aabf96-b1fe-4c82-af00-e9e034e69e14\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:51.814870 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:51.814811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:52.057352 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.057319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l" event={"ID":"16b4f397-7fdf-4223-9c6a-3100c148db24","Type":"ContainerStarted","Data":"64e0df7e1b83e1df0426d01717673638c463166686657199f30877a8c60bad57"}
Sep 11 03:55:52.089645 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.089612    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"]
Sep 11 03:55:52.092130 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:55:52.092102    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod99aabf96_b1fe_4c82_af00_e9e034e69e14.slice/crio-f9928821c74da336513f48550ee3765dec3da96d62ddd67c1cf007d62db5267c WatchSource:0}: Error finding container f9928821c74da336513f48550ee3765dec3da96d62ddd67c1cf007d62db5267c: Status 404 returned error can't find the container with id f9928821c74da336513f48550ee3765dec3da96d62ddd67c1cf007d62db5267c
Sep 11 03:55:52.429430 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.429328    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"]
Sep 11 03:55:52.429615 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.429574    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m" podUID="88f845e7-50f6-4d23-8bfb-48ad246ee505" containerName="discovery" containerID="cri-o://871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865" gracePeriod=30
Sep 11 03:55:52.722064 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.722025    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:52.778260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.777718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-dns-cert\") pod \"88f845e7-50f6-4d23-8bfb-48ad246ee505\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") "
Sep 11 03:55:52.778260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.777777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-ca-configmap\") pod \"88f845e7-50f6-4d23-8bfb-48ad246ee505\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") "
Sep 11 03:55:52.778260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.777809    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-kubeconfig\") pod \"88f845e7-50f6-4d23-8bfb-48ad246ee505\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") "
Sep 11 03:55:52.778260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.777834    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5hld\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-kube-api-access-c5hld\") pod \"88f845e7-50f6-4d23-8bfb-48ad246ee505\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") "
Sep 11 03:55:52.778260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.777880    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f845e7-50f6-4d23-8bfb-48ad246ee505-local-certs\") pod \"88f845e7-50f6-4d23-8bfb-48ad246ee505\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") "
Sep 11 03:55:52.778260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.777917    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-cacerts\") pod \"88f845e7-50f6-4d23-8bfb-48ad246ee505\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") "
Sep 11 03:55:52.778260 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.777939    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-token\") pod \"88f845e7-50f6-4d23-8bfb-48ad246ee505\" (UID: \"88f845e7-50f6-4d23-8bfb-48ad246ee505\") "
Sep 11 03:55:52.778764 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.778505    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "88f845e7-50f6-4d23-8bfb-48ad246ee505" (UID: "88f845e7-50f6-4d23-8bfb-48ad246ee505"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:55:52.781246 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.781209    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-kube-api-access-c5hld" (OuterVolumeSpecName: "kube-api-access-c5hld") pod "88f845e7-50f6-4d23-8bfb-48ad246ee505" (UID: "88f845e7-50f6-4d23-8bfb-48ad246ee505"). InnerVolumeSpecName "kube-api-access-c5hld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:52.783828 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.783788    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f845e7-50f6-4d23-8bfb-48ad246ee505-local-certs" (OuterVolumeSpecName: "local-certs") pod "88f845e7-50f6-4d23-8bfb-48ad246ee505" (UID: "88f845e7-50f6-4d23-8bfb-48ad246ee505"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:55:52.783828 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.783801    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "88f845e7-50f6-4d23-8bfb-48ad246ee505" (UID: "88f845e7-50f6-4d23-8bfb-48ad246ee505"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:52.784273 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.784073    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "88f845e7-50f6-4d23-8bfb-48ad246ee505" (UID: "88f845e7-50f6-4d23-8bfb-48ad246ee505"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:52.784273 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.784247    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-token" (OuterVolumeSpecName: "istio-token") pod "88f845e7-50f6-4d23-8bfb-48ad246ee505" (UID: "88f845e7-50f6-4d23-8bfb-48ad246ee505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:55:52.784681 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.784647    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-cacerts" (OuterVolumeSpecName: "cacerts") pod "88f845e7-50f6-4d23-8bfb-48ad246ee505" (UID: "88f845e7-50f6-4d23-8bfb-48ad246ee505"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:55:52.879554 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.879520    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-ca-configmap\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:55:52.879554 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.879555    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-kubeconfig\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:55:52.879783 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.879571    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5hld\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-kube-api-access-c5hld\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:55:52.879783 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.879581    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f845e7-50f6-4d23-8bfb-48ad246ee505-local-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:55:52.879783 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.879590    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-cacerts\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:55:52.879783 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.879597    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:55:52.879783 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:52.879605    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/88f845e7-50f6-4d23-8bfb-48ad246ee505-istio-csr-dns-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:55:53.065261 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.065226    2566 generic.go:356] "Generic (PLEG): container finished" podID="88f845e7-50f6-4d23-8bfb-48ad246ee505" containerID="871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865" exitCode=0
Sep 11 03:55:53.065693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.065285    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"
Sep 11 03:55:53.065693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.065308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m" event={"ID":"88f845e7-50f6-4d23-8bfb-48ad246ee505","Type":"ContainerDied","Data":"871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865"}
Sep 11 03:55:53.065693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.065369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m" event={"ID":"88f845e7-50f6-4d23-8bfb-48ad246ee505","Type":"ContainerDied","Data":"d74641b9e0436e51bcb795c97e95348d209560d528319bfe471fa929b197686a"}
Sep 11 03:55:53.065693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.065401    2566 scope.go:117] "RemoveContainer" containerID="871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865"
Sep 11 03:55:53.067813 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.067777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc" event={"ID":"99aabf96-b1fe-4c82-af00-e9e034e69e14","Type":"ContainerStarted","Data":"f9928821c74da336513f48550ee3765dec3da96d62ddd67c1cf007d62db5267c"}
Sep 11 03:55:53.092528 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.092431    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"]
Sep 11 03:55:53.095312 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.095291    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9cg2m"]
Sep 11 03:55:53.854057 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.853880    2566 scope.go:117] "RemoveContainer" containerID="871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865"
Sep 11 03:55:53.854266 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:55:53.854246    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865\": container with ID starting with 871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865 not found: ID does not exist" containerID="871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865"
Sep 11 03:55:53.854322 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:53.854277    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865"} err="failed to get container status \"871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865\": rpc error: code = NotFound desc = could not find container \"871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865\": container with ID starting with 871778b14e3f7141c4be06972706957e603cb9cdc3171605750ae51237dc6865 not found: ID does not exist"
Sep 11 03:55:54.611332 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:54.611305    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f845e7-50f6-4d23-8bfb-48ad246ee505" path="/var/lib/kubelet/pods/88f845e7-50f6-4d23-8bfb-48ad246ee505/volumes"
Sep 11 03:55:55.080057 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:55.079967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc" event={"ID":"99aabf96-b1fe-4c82-af00-e9e034e69e14","Type":"ContainerStarted","Data":"e70bc937f804a52b0b094839a703c726b7ee4892442c5ef0e30260524c02b8d2"}
Sep 11 03:55:55.080057 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:55.080046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:55:55.081563 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:55.081537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l" event={"ID":"16b4f397-7fdf-4223-9c6a-3100c148db24","Type":"ContainerStarted","Data":"d4d5fcd1c4949025eac4b3a82845790296bb4d26f1ee57a4e374cc607eaf8d86"}
Sep 11 03:55:55.081693 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:55.081675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:55:55.096479 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:55.096401    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc" podStartSLOduration=2.630339517 podStartE2EDuration="5.096383662s" totalImagesPullingTime="2.466044145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:55.095939314 +0000 UTC m=+352.961959050" watchObservedRunningTime="2026-09-11 03:55:55.096383662 +0000 UTC m=+352.962403400"
Sep 11 03:55:55.114822 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:55.114772    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l" podStartSLOduration=2.104188972 podStartE2EDuration="5.114759415s" totalImagesPullingTime="3.010570443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:55.112945829 +0000 UTC m=+352.978965564" watchObservedRunningTime="2026-09-11 03:55:55.114759415 +0000 UTC m=+352.980779151"
Sep 11 03:55:57.039904 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:55:57.039872    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4kqzt"
Sep 11 03:56:26.088723 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:26.088692    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-t6kbc"
Sep 11 03:56:26.091764 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:26.091741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gt2l"
Sep 11 03:56:59.880862 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.880827    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"]
Sep 11 03:56:59.881276 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.881098    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f845e7-50f6-4d23-8bfb-48ad246ee505" containerName="discovery"
Sep 11 03:56:59.881276 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.881110    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f845e7-50f6-4d23-8bfb-48ad246ee505" containerName="discovery"
Sep 11 03:56:59.881276 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.881155    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f845e7-50f6-4d23-8bfb-48ad246ee505" containerName="discovery"
Sep 11 03:56:59.883965 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.883948    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:56:59.886464 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.886437    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 03:56:59.886591 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.886578    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 03:56:59.886651 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.886624    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-762n8\""
Sep 11 03:56:59.898111 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.898089    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"]
Sep 11 03:56:59.961468 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.961442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5cx\" (UniqueName: \"kubernetes.io/projected/97c7c5dd-b774-4068-b4cd-c50bc738d46b-kube-api-access-tq5cx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" (UID: \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:56:59.961587 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:56:59.961507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97c7c5dd-b774-4068-b4cd-c50bc738d46b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" (UID: \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:00.062036 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:00.062000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97c7c5dd-b774-4068-b4cd-c50bc738d46b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" (UID: \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:00.062149 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:00.062062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5cx\" (UniqueName: \"kubernetes.io/projected/97c7c5dd-b774-4068-b4cd-c50bc738d46b-kube-api-access-tq5cx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" (UID: \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:00.062378 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:00.062358    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97c7c5dd-b774-4068-b4cd-c50bc738d46b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" (UID: \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:00.070601 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:00.070579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5cx\" (UniqueName: \"kubernetes.io/projected/97c7c5dd-b774-4068-b4cd-c50bc738d46b-kube-api-access-tq5cx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" (UID: \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:00.194112 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:00.194061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:00.482981 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:00.482960    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"]
Sep 11 03:57:00.485174 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:57:00.485147    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c7c5dd_b774_4068_b4cd_c50bc738d46b.slice/crio-0505733fc634b88a29374fd751ac339bb0ad63cf4b91296a3ae449c387d188d6 WatchSource:0}: Error finding container 0505733fc634b88a29374fd751ac339bb0ad63cf4b91296a3ae449c387d188d6: Status 404 returned error can't find the container with id 0505733fc634b88a29374fd751ac339bb0ad63cf4b91296a3ae449c387d188d6
Sep 11 03:57:01.346938 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:01.346895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj" event={"ID":"97c7c5dd-b774-4068-b4cd-c50bc738d46b","Type":"ContainerStarted","Data":"0505733fc634b88a29374fd751ac339bb0ad63cf4b91296a3ae449c387d188d6"}
Sep 11 03:57:01.883813 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:01.883780    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"]
Sep 11 03:57:01.887348 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:01.887320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:01.890127 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:01.889953    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-st7dj\""
Sep 11 03:57:01.897771 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:01.897254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"]
Sep 11 03:57:01.983422 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:01.983369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgvjk\" (UniqueName: \"kubernetes.io/projected/5e335cb5-4258-44fe-8394-ec18867e7b5b-kube-api-access-hgvjk\") pod \"limitador-operator-controller-manager-74495c69f8-cnhl9\" (UID: \"5e335cb5-4258-44fe-8394-ec18867e7b5b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:02.084324 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:02.084282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgvjk\" (UniqueName: \"kubernetes.io/projected/5e335cb5-4258-44fe-8394-ec18867e7b5b-kube-api-access-hgvjk\") pod \"limitador-operator-controller-manager-74495c69f8-cnhl9\" (UID: \"5e335cb5-4258-44fe-8394-ec18867e7b5b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:02.095576 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:02.095517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgvjk\" (UniqueName: \"kubernetes.io/projected/5e335cb5-4258-44fe-8394-ec18867e7b5b-kube-api-access-hgvjk\") pod \"limitador-operator-controller-manager-74495c69f8-cnhl9\" (UID: \"5e335cb5-4258-44fe-8394-ec18867e7b5b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:02.201338 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:02.201261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:03.051116 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:03.051093    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"]
Sep 11 03:57:03.053240 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:57:03.053212    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e335cb5_4258_44fe_8394_ec18867e7b5b.slice/crio-4b6535a727ce5e01e61e32c69de3f67700067048818a9f86f71831a4d543f3c1 WatchSource:0}: Error finding container 4b6535a727ce5e01e61e32c69de3f67700067048818a9f86f71831a4d543f3c1: Status 404 returned error can't find the container with id 4b6535a727ce5e01e61e32c69de3f67700067048818a9f86f71831a4d543f3c1
Sep 11 03:57:03.358336 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:03.358243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9" event={"ID":"5e335cb5-4258-44fe-8394-ec18867e7b5b","Type":"ContainerStarted","Data":"4b6535a727ce5e01e61e32c69de3f67700067048818a9f86f71831a4d543f3c1"}
Sep 11 03:57:04.582697 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:04.582664    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fpjqt"]
Sep 11 03:57:04.586150 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:04.586128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt"
Sep 11 03:57:04.590537 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:04.590515    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x8vzg\""
Sep 11 03:57:04.597489 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:04.597464    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fpjqt"]
Sep 11 03:57:04.707930 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:04.707894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjdfp\" (UniqueName: \"kubernetes.io/projected/5282db95-0b76-4416-9840-34d992e4ef3a-kube-api-access-cjdfp\") pod \"authorino-operator-76b478bd78-fpjqt\" (UID: \"5282db95-0b76-4416-9840-34d992e4ef3a\") " pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt"
Sep 11 03:57:04.809269 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:04.809190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjdfp\" (UniqueName: \"kubernetes.io/projected/5282db95-0b76-4416-9840-34d992e4ef3a-kube-api-access-cjdfp\") pod \"authorino-operator-76b478bd78-fpjqt\" (UID: \"5282db95-0b76-4416-9840-34d992e4ef3a\") " pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt"
Sep 11 03:57:04.820606 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:04.820575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjdfp\" (UniqueName: \"kubernetes.io/projected/5282db95-0b76-4416-9840-34d992e4ef3a-kube-api-access-cjdfp\") pod \"authorino-operator-76b478bd78-fpjqt\" (UID: \"5282db95-0b76-4416-9840-34d992e4ef3a\") " pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt"
Sep 11 03:57:04.897400 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:04.897368    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt"
Sep 11 03:57:05.218428 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:05.218372    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fpjqt"]
Sep 11 03:57:05.239371 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:57:05.239336    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5282db95_0b76_4416_9840_34d992e4ef3a.slice/crio-9c796c9cbd36f0d2fcb718dd7b1f06b9deb7e2719acb0763c8ce301f6ff86fa7 WatchSource:0}: Error finding container 9c796c9cbd36f0d2fcb718dd7b1f06b9deb7e2719acb0763c8ce301f6ff86fa7: Status 404 returned error can't find the container with id 9c796c9cbd36f0d2fcb718dd7b1f06b9deb7e2719acb0763c8ce301f6ff86fa7
Sep 11 03:57:05.369294 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:05.369259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj" event={"ID":"97c7c5dd-b774-4068-b4cd-c50bc738d46b","Type":"ContainerStarted","Data":"724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994"}
Sep 11 03:57:05.369466 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:05.369449    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:05.371048 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:05.371023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9" event={"ID":"5e335cb5-4258-44fe-8394-ec18867e7b5b","Type":"ContainerStarted","Data":"0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e"}
Sep 11 03:57:05.371121 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:05.371080    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:05.372325 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:05.372307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt" event={"ID":"5282db95-0b76-4416-9840-34d992e4ef3a","Type":"ContainerStarted","Data":"9c796c9cbd36f0d2fcb718dd7b1f06b9deb7e2719acb0763c8ce301f6ff86fa7"}
Sep 11 03:57:05.390003 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:05.389918    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj" podStartSLOduration=2.381137852 podStartE2EDuration="6.389904899s" totalImagesPullingTime="4.008767047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:05.389011753 +0000 UTC m=+423.255031488" watchObservedRunningTime="2026-09-11 03:57:05.389904899 +0000 UTC m=+423.255924635"
Sep 11 03:57:05.412687 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:05.412645    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9" podStartSLOduration=2.17224314 podStartE2EDuration="4.412630672s" totalImagesPullingTime="2.240387532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:05.412043875 +0000 UTC m=+423.278063617" watchObservedRunningTime="2026-09-11 03:57:05.412630672 +0000 UTC m=+423.278650423"
Sep 11 03:57:08.386522 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:08.386489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt" event={"ID":"5282db95-0b76-4416-9840-34d992e4ef3a","Type":"ContainerStarted","Data":"d3edabf002372bb50f6d62cdb9cf14ff40b1fd843c3f708b25d443d70124f60a"}
Sep 11 03:57:08.386929 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:08.386627    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt"
Sep 11 03:57:08.403192 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:08.403147    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt" podStartSLOduration=1.939018932 podStartE2EDuration="4.403133351s" totalImagesPullingTime="2.464114419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:08.40166909 +0000 UTC m=+426.267688822" watchObservedRunningTime="2026-09-11 03:57:08.403133351 +0000 UTC m=+426.269153086"
Sep 11 03:57:16.378530 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:16.378496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:16.378948 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:16.378652    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:18.618849 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.618815    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"]
Sep 11 03:57:18.619221 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.619117    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" containerName="manager" containerID="cri-o://724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994" gracePeriod=2
Sep 11 03:57:18.621173 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.621137    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:18.621749 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.621699    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"]
Sep 11 03:57:18.635133 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.635113    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"]
Sep 11 03:57:18.635369 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.635333    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" containerName="manager" containerID="cri-o://0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e" gracePeriod=2
Sep 11 03:57:18.637632 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.637595    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:18.644828 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.644774    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"]
Sep 11 03:57:18.645688 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.645667    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" containerName="manager"
Sep 11 03:57:18.645839 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.645825    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" containerName="manager"
Sep 11 03:57:18.646153 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.646133    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" containerName="manager"
Sep 11 03:57:18.652815 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.650612    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"]
Sep 11 03:57:18.652815 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.650749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:18.658359 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.658027    2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"5e335cb5-4258-44fe-8394-ec18867e7b5b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-cnhl9\": pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:18.659954 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.659920    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"]
Sep 11 03:57:18.660337 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.660314    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" containerName="manager"
Sep 11 03:57:18.660337 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.660338    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" containerName="manager"
Sep 11 03:57:18.660503 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.660329    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:18.660503 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.660428    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" containerName="manager"
Sep 11 03:57:18.662012 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.661982    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:18.663322 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.663290    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"]
Sep 11 03:57:18.663453 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.663432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"
Sep 11 03:57:18.677947 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.677924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"]
Sep 11 03:57:18.712652 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.712624    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:18.714698 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.714677    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:18.814030 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.813992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pvz67\" (UID: \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:18.814138 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.814084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp6xb\" (UniqueName: \"kubernetes.io/projected/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-kube-api-access-fp6xb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pvz67\" (UID: \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:18.814177 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.814127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwz72\" (UniqueName: \"kubernetes.io/projected/4d6a69fa-0ff7-45df-ae28-48c109de7e9d-kube-api-access-pwz72\") pod \"limitador-operator-controller-manager-74495c69f8-c526b\" (UID: \"4d6a69fa-0ff7-45df-ae28-48c109de7e9d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"
Sep 11 03:57:18.862564 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.862532    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:18.864893 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.864870    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:18.864991 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.864939    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:18.867254 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.867232    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:18.869597 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.869556    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:18.872194 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:18.872177    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:18.914904 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.914878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pvz67\" (UID: \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:18.915005 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.914943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp6xb\" (UniqueName: \"kubernetes.io/projected/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-kube-api-access-fp6xb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pvz67\" (UID: \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:18.915005 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.914972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwz72\" (UniqueName: \"kubernetes.io/projected/4d6a69fa-0ff7-45df-ae28-48c109de7e9d-kube-api-access-pwz72\") pod \"limitador-operator-controller-manager-74495c69f8-c526b\" (UID: \"4d6a69fa-0ff7-45df-ae28-48c109de7e9d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"
Sep 11 03:57:18.915249 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.915227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pvz67\" (UID: \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:18.923634 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.923615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp6xb\" (UniqueName: \"kubernetes.io/projected/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-kube-api-access-fp6xb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pvz67\" (UID: \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:18.923821 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:18.923804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwz72\" (UniqueName: \"kubernetes.io/projected/4d6a69fa-0ff7-45df-ae28-48c109de7e9d-kube-api-access-pwz72\") pod \"limitador-operator-controller-manager-74495c69f8-c526b\" (UID: \"4d6a69fa-0ff7-45df-ae28-48c109de7e9d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"
Sep 11 03:57:19.015978 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.015948    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq5cx\" (UniqueName: \"kubernetes.io/projected/97c7c5dd-b774-4068-b4cd-c50bc738d46b-kube-api-access-tq5cx\") pod \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\" (UID: \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\") "
Sep 11 03:57:19.016067 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.016016    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgvjk\" (UniqueName: \"kubernetes.io/projected/5e335cb5-4258-44fe-8394-ec18867e7b5b-kube-api-access-hgvjk\") pod \"5e335cb5-4258-44fe-8394-ec18867e7b5b\" (UID: \"5e335cb5-4258-44fe-8394-ec18867e7b5b\") "
Sep 11 03:57:19.016067 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.016049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97c7c5dd-b774-4068-b4cd-c50bc738d46b-extensions-socket-volume\") pod \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\" (UID: \"97c7c5dd-b774-4068-b4cd-c50bc738d46b\") "
Sep 11 03:57:19.016533 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.016510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c7c5dd-b774-4068-b4cd-c50bc738d46b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "97c7c5dd-b774-4068-b4cd-c50bc738d46b" (UID: "97c7c5dd-b774-4068-b4cd-c50bc738d46b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:19.017768 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.017746    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e335cb5-4258-44fe-8394-ec18867e7b5b-kube-api-access-hgvjk" (OuterVolumeSpecName: "kube-api-access-hgvjk") pod "5e335cb5-4258-44fe-8394-ec18867e7b5b" (UID: "5e335cb5-4258-44fe-8394-ec18867e7b5b"). InnerVolumeSpecName "kube-api-access-hgvjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:19.017845 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.017778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c7c5dd-b774-4068-b4cd-c50bc738d46b-kube-api-access-tq5cx" (OuterVolumeSpecName: "kube-api-access-tq5cx") pod "97c7c5dd-b774-4068-b4cd-c50bc738d46b" (UID: "97c7c5dd-b774-4068-b4cd-c50bc738d46b"). InnerVolumeSpecName "kube-api-access-tq5cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:19.058675 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.058653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:19.064511 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.064496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"
Sep 11 03:57:19.118986 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.118117    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgvjk\" (UniqueName: \"kubernetes.io/projected/5e335cb5-4258-44fe-8394-ec18867e7b5b-kube-api-access-hgvjk\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:57:19.118986 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.118145    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97c7c5dd-b774-4068-b4cd-c50bc738d46b-extensions-socket-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:57:19.118986 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.118160    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq5cx\" (UniqueName: \"kubernetes.io/projected/97c7c5dd-b774-4068-b4cd-c50bc738d46b-kube-api-access-tq5cx\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:57:19.340563 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.340531    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"]
Sep 11 03:57:19.341578 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.341552    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"]
Sep 11 03:57:19.344416 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:57:19.344379    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6a69fa_0ff7_45df_ae28_48c109de7e9d.slice/crio-e297af68643a46109971f37469935da0d2b4c5780e75813e73ec3cf3122fc706 WatchSource:0}: Error finding container e297af68643a46109971f37469935da0d2b4c5780e75813e73ec3cf3122fc706: Status 404 returned error can't find the container with id e297af68643a46109971f37469935da0d2b4c5780e75813e73ec3cf3122fc706
Sep 11 03:57:19.344914 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:57:19.344894    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec5706a_8d47_4f9c_87d6_ac36dd51fcee.slice/crio-da7a2193fd82f15dd1c2ea248ea7d651dd6cac1e6898e323185d4e786bacf857 WatchSource:0}: Error finding container da7a2193fd82f15dd1c2ea248ea7d651dd6cac1e6898e323185d4e786bacf857: Status 404 returned error can't find the container with id da7a2193fd82f15dd1c2ea248ea7d651dd6cac1e6898e323185d4e786bacf857
Sep 11 03:57:19.393342 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.393324    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-fpjqt"
Sep 11 03:57:19.397026 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.397002    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:19.399024 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.398994    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:19.437700 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.437671    2566 generic.go:356] "Generic (PLEG): container finished" podID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" containerID="724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994" exitCode=0
Sep 11 03:57:19.437835 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.437715    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:19.437835 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.437781    2566 scope.go:117] "RemoveContainer" containerID="724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994"
Sep 11 03:57:19.439520 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.439500    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e335cb5-4258-44fe-8394-ec18867e7b5b" containerID="0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e" exitCode=0
Sep 11 03:57:19.439661 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.439592    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:19.440185 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.440088    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:19.442818 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.442510    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:19.444116 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.443700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b" event={"ID":"4d6a69fa-0ff7-45df-ae28-48c109de7e9d","Type":"ContainerStarted","Data":"9fb72478e72bd034eef7ab147e0efe818edfbdcba3b54e7d76aa4d10c4e09211"}
Sep 11 03:57:19.444116 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.443758    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"
Sep 11 03:57:19.444116 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.443784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b" event={"ID":"4d6a69fa-0ff7-45df-ae28-48c109de7e9d","Type":"ContainerStarted","Data":"e297af68643a46109971f37469935da0d2b4c5780e75813e73ec3cf3122fc706"}
Sep 11 03:57:19.448228 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.445583    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:19.448228 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.446937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67" event={"ID":"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee","Type":"ContainerStarted","Data":"eb97e0c1ca87317707cbd17ae42b03bcbff39423489235216966b6a14ad0b8c2"}
Sep 11 03:57:19.448228 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.446978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67" event={"ID":"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee","Type":"ContainerStarted","Data":"da7a2193fd82f15dd1c2ea248ea7d651dd6cac1e6898e323185d4e786bacf857"}
Sep 11 03:57:19.448228 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.447619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:19.448228 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.447768    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:19.448228 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.447953    2566 scope.go:117] "RemoveContainer" containerID="724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994"
Sep 11 03:57:19.448828 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.448266    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994\": container with ID starting with 724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994 not found: ID does not exist" containerID="724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994"
Sep 11 03:57:19.448828 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.448295    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994"} err="failed to get container status \"724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994\": rpc error: code = NotFound desc = could not find container \"724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994\": container with ID starting with 724052d097016868d653d3c95f23e750a3ce43f35d44bd4c1db3ed581d7f7994 not found: ID does not exist"
Sep 11 03:57:19.448828 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.448316    2566 scope.go:117] "RemoveContainer" containerID="0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e"
Sep 11 03:57:19.449956 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.449915    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:19.452115 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.451895    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:19.458178 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.458159    2566 scope.go:117] "RemoveContainer" containerID="0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e"
Sep 11 03:57:19.458427 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.458395    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e\": container with ID starting with 0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e not found: ID does not exist" containerID="0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e"
Sep 11 03:57:19.458510 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.458435    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e"} err="failed to get container status \"0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e\": rpc error: code = NotFound desc = could not find container \"0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e\": container with ID starting with 0ceeffcca10533f21e32a300b073c224f352e0e574feec41c197b574d3ab003e not found: ID does not exist"
Sep 11 03:57:19.474832 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.474795    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67" podStartSLOduration=1.474783427 podStartE2EDuration="1.474783427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:19.472937882 +0000 UTC m=+437.338957617" watchObservedRunningTime="2026-09-11 03:57:19.474783427 +0000 UTC m=+437.340803162"
Sep 11 03:57:19.494941 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:19.494900    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b" podStartSLOduration=1.494886709 podStartE2EDuration="1.494886709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:19.49426493 +0000 UTC m=+437.360284666" watchObservedRunningTime="2026-09-11 03:57:19.494886709 +0000 UTC m=+437.360906456"
Sep 11 03:57:19.496529 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.496506    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gqhnj\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gqhnj"
Sep 11 03:57:19.498586 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:57:19.498567    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-cnhl9\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cnhl9"
Sep 11 03:57:20.611923 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:20.611891    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e335cb5-4258-44fe-8394-ec18867e7b5b" path="/var/lib/kubelet/pods/5e335cb5-4258-44fe-8394-ec18867e7b5b/volumes"
Sep 11 03:57:20.612247 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:20.612233    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c7c5dd-b774-4068-b4cd-c50bc738d46b" path="/var/lib/kubelet/pods/97c7c5dd-b774-4068-b4cd-c50bc738d46b/volumes"
Sep 11 03:57:30.456587 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:30.456558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-c526b"
Sep 11 03:57:31.461668 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:31.461642    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:36.918112 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:36.918077    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"]
Sep 11 03:57:36.918508 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:36.918274    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67" podUID="0ec5706a-8d47-4f9c-87d6-ac36dd51fcee" containerName="manager" containerID="cri-o://eb97e0c1ca87317707cbd17ae42b03bcbff39423489235216966b6a14ad0b8c2" gracePeriod=10
Sep 11 03:57:37.527956 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:37.527924    2566 generic.go:356] "Generic (PLEG): container finished" podID="0ec5706a-8d47-4f9c-87d6-ac36dd51fcee" containerID="eb97e0c1ca87317707cbd17ae42b03bcbff39423489235216966b6a14ad0b8c2" exitCode=0
Sep 11 03:57:37.528104 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:37.527995    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67" event={"ID":"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee","Type":"ContainerDied","Data":"eb97e0c1ca87317707cbd17ae42b03bcbff39423489235216966b6a14ad0b8c2"}
Sep 11 03:57:37.885942 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:37.885918    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:38.049102 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.049041    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-extensions-socket-volume\") pod \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\" (UID: \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\") "
Sep 11 03:57:38.049102 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.049072    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp6xb\" (UniqueName: \"kubernetes.io/projected/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-kube-api-access-fp6xb\") pod \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\" (UID: \"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee\") "
Sep 11 03:57:38.049477 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.049350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0ec5706a-8d47-4f9c-87d6-ac36dd51fcee" (UID: "0ec5706a-8d47-4f9c-87d6-ac36dd51fcee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 03:57:38.050980 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.050953    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-kube-api-access-fp6xb" (OuterVolumeSpecName: "kube-api-access-fp6xb") pod "0ec5706a-8d47-4f9c-87d6-ac36dd51fcee" (UID: "0ec5706a-8d47-4f9c-87d6-ac36dd51fcee"). InnerVolumeSpecName "kube-api-access-fp6xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:57:38.150154 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.150134    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-extensions-socket-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:57:38.150154 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.150156    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp6xb\" (UniqueName: \"kubernetes.io/projected/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee-kube-api-access-fp6xb\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:57:38.533738 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.533717    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"
Sep 11 03:57:38.533738 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.533730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67" event={"ID":"0ec5706a-8d47-4f9c-87d6-ac36dd51fcee","Type":"ContainerDied","Data":"da7a2193fd82f15dd1c2ea248ea7d651dd6cac1e6898e323185d4e786bacf857"}
Sep 11 03:57:38.533893 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.533782    2566 scope.go:117] "RemoveContainer" containerID="eb97e0c1ca87317707cbd17ae42b03bcbff39423489235216966b6a14ad0b8c2"
Sep 11 03:57:38.554239 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.554220    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"]
Sep 11 03:57:38.559538 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.559516    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pvz67"]
Sep 11 03:57:38.611937 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:38.611911    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec5706a-8d47-4f9c-87d6-ac36dd51fcee" path="/var/lib/kubelet/pods/0ec5706a-8d47-4f9c-87d6-ac36dd51fcee/volumes"
Sep 11 03:57:53.173989 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.173961    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"]
Sep 11 03:57:53.174540 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.174260    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec5706a-8d47-4f9c-87d6-ac36dd51fcee" containerName="manager"
Sep 11 03:57:53.174540 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.174272    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec5706a-8d47-4f9c-87d6-ac36dd51fcee" containerName="manager"
Sep 11 03:57:53.174540 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.174349    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec5706a-8d47-4f9c-87d6-ac36dd51fcee" containerName="manager"
Sep 11 03:57:53.178616 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.178593    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.181035 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.181014    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 03:57:53.181136 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.181069    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7ln89\""
Sep 11 03:57:53.194191 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.194169    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"]
Sep 11 03:57:53.258643 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.258754 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.258754 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkp2g\" (UniqueName: \"kubernetes.io/projected/2028fff4-7d56-4cee-be2c-0e1e054793a5-kube-api-access-rkp2g\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.258754 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.258894 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.258894 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.258894 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.258894 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2028fff4-7d56-4cee-be2c-0e1e054793a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.258894 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.258886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.359577 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.359556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.359682 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.359589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.359682 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.359610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.359682 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.359632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.359799 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.359754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2028fff4-7d56-4cee-be2c-0e1e054793a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.359844 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.359798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.359844 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.359840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.359990 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.359967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.360136 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.360012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.360291 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.360255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkp2g\" (UniqueName: \"kubernetes.io/projected/2028fff4-7d56-4cee-be2c-0e1e054793a5-kube-api-access-rkp2g\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.360387 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.360146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.360387 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.360153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.360387 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.360354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.360527 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.360491    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2028fff4-7d56-4cee-be2c-0e1e054793a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.361844 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.361826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.362074 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.362058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.367562 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.367539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkp2g\" (UniqueName: \"kubernetes.io/projected/2028fff4-7d56-4cee-be2c-0e1e054793a5-kube-api-access-rkp2g\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.368075 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.368052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2028fff4-7d56-4cee-be2c-0e1e054793a5-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nmjcb\" (UID: \"2028fff4-7d56-4cee-be2c-0e1e054793a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.490359 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.490338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:53.761700 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:53.761672    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"]
Sep 11 03:57:53.763701 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:57:53.763662    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2028fff4_7d56_4cee_be2c_0e1e054793a5.slice/crio-591fa67342a3268ced2ea61dec30e30b5aa64362e1da3e7c3b6e9fb4a582f112 WatchSource:0}: Error finding container 591fa67342a3268ced2ea61dec30e30b5aa64362e1da3e7c3b6e9fb4a582f112: Status 404 returned error can't find the container with id 591fa67342a3268ced2ea61dec30e30b5aa64362e1da3e7c3b6e9fb4a582f112
Sep 11 03:57:54.601469 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:54.601433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" event={"ID":"2028fff4-7d56-4cee-be2c-0e1e054793a5","Type":"ContainerStarted","Data":"591fa67342a3268ced2ea61dec30e30b5aa64362e1da3e7c3b6e9fb4a582f112"}
Sep 11 03:57:58.089709 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:58.089676    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:57:58.089975 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:58.089747    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:57:58.089975 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:58.089773    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:57:58.622499 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:58.622461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" event={"ID":"2028fff4-7d56-4cee-be2c-0e1e054793a5","Type":"ContainerStarted","Data":"6ff40a712c54f71285c70c7c99c104660be8d34c330c88be5e87a4fc084e829d"}
Sep 11 03:57:58.642469 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:58.642421    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" podStartSLOduration=1.318574655 podStartE2EDuration="5.642390999s" totalImagesPullingTime="4.323816344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:57:58.64027333 +0000 UTC m=+476.506293067" watchObservedRunningTime="2026-09-11 03:57:58.642390999 +0000 UTC m=+476.508410733"
Sep 11 03:57:59.490815 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.490785    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:59.495077 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.495054    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:59.598594 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.598565    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfgx8"]
Sep 11 03:57:59.601748 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.601724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:57:59.604319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.604295    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7dpgb\""
Sep 11 03:57:59.604420 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.604358    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 03:57:59.616115 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.616097    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfgx8"]
Sep 11 03:57:59.627259 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.627239    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:59.628168 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.628149    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:57:59.696078 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.696049    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfgx8"]
Sep 11 03:57:59.705513 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.705479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/acd2b8a0-5f53-4cac-b56c-ea05f1896497-config-file\") pod \"limitador-limitador-8588749977-wfgx8\" (UID: \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\") " pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:57:59.705652 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.705563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh25w\" (UniqueName: \"kubernetes.io/projected/acd2b8a0-5f53-4cac-b56c-ea05f1896497-kube-api-access-hh25w\") pod \"limitador-limitador-8588749977-wfgx8\" (UID: \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\") " pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:57:59.806771 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.806681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/acd2b8a0-5f53-4cac-b56c-ea05f1896497-config-file\") pod \"limitador-limitador-8588749977-wfgx8\" (UID: \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\") " pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:57:59.806938 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.806858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh25w\" (UniqueName: \"kubernetes.io/projected/acd2b8a0-5f53-4cac-b56c-ea05f1896497-kube-api-access-hh25w\") pod \"limitador-limitador-8588749977-wfgx8\" (UID: \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\") " pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:57:59.807431 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.807385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/acd2b8a0-5f53-4cac-b56c-ea05f1896497-config-file\") pod \"limitador-limitador-8588749977-wfgx8\" (UID: \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\") " pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:57:59.817725 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.817704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh25w\" (UniqueName: \"kubernetes.io/projected/acd2b8a0-5f53-4cac-b56c-ea05f1896497-kube-api-access-hh25w\") pod \"limitador-limitador-8588749977-wfgx8\" (UID: \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\") " pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:57:59.911720 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.911694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:57:59.987706 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.987671    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 03:57:59.992025 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:57:59.992009    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:00.008008 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.007982    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 03:58:00.061364 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.061297    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 03:58:00.109361 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.109338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84a96ab1-cd48-409d-90bc-7af13fbcea18-config-file\") pod \"limitador-limitador-84994f7ffc-87jlk\" (UID: \"84a96ab1-cd48-409d-90bc-7af13fbcea18\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:00.109525 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.109389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c42s\" (UniqueName: \"kubernetes.io/projected/84a96ab1-cd48-409d-90bc-7af13fbcea18-kube-api-access-9c42s\") pod \"limitador-limitador-84994f7ffc-87jlk\" (UID: \"84a96ab1-cd48-409d-90bc-7af13fbcea18\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:00.177945 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.177924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfgx8"]
Sep 11 03:58:00.180500 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:58:00.180468    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd2b8a0_5f53_4cac_b56c_ea05f1896497.slice/crio-f0f96aad1947c38b3b98747172916c9b47974e87da52dcd7d97b9fda4eef5457 WatchSource:0}: Error finding container f0f96aad1947c38b3b98747172916c9b47974e87da52dcd7d97b9fda4eef5457: Status 404 returned error can't find the container with id f0f96aad1947c38b3b98747172916c9b47974e87da52dcd7d97b9fda4eef5457
Sep 11 03:58:00.210857 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.210833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84a96ab1-cd48-409d-90bc-7af13fbcea18-config-file\") pod \"limitador-limitador-84994f7ffc-87jlk\" (UID: \"84a96ab1-cd48-409d-90bc-7af13fbcea18\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:00.210938 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.210868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c42s\" (UniqueName: \"kubernetes.io/projected/84a96ab1-cd48-409d-90bc-7af13fbcea18-kube-api-access-9c42s\") pod \"limitador-limitador-84994f7ffc-87jlk\" (UID: \"84a96ab1-cd48-409d-90bc-7af13fbcea18\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:00.211424 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.211390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84a96ab1-cd48-409d-90bc-7af13fbcea18-config-file\") pod \"limitador-limitador-84994f7ffc-87jlk\" (UID: \"84a96ab1-cd48-409d-90bc-7af13fbcea18\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:00.219484 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.219462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c42s\" (UniqueName: \"kubernetes.io/projected/84a96ab1-cd48-409d-90bc-7af13fbcea18-kube-api-access-9c42s\") pod \"limitador-limitador-84994f7ffc-87jlk\" (UID: \"84a96ab1-cd48-409d-90bc-7af13fbcea18\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:00.302152 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.302128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:00.567729 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.567707    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 03:58:00.570033 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:58:00.570004    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a96ab1_cd48_409d_90bc_7af13fbcea18.slice/crio-97f8ffa41f381807712573cdd9de19a324094381a82bc618f9e2a7a28ef97479 WatchSource:0}: Error finding container 97f8ffa41f381807712573cdd9de19a324094381a82bc618f9e2a7a28ef97479: Status 404 returned error can't find the container with id 97f8ffa41f381807712573cdd9de19a324094381a82bc618f9e2a7a28ef97479
Sep 11 03:58:00.632453 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.632379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk" event={"ID":"84a96ab1-cd48-409d-90bc-7af13fbcea18","Type":"ContainerStarted","Data":"97f8ffa41f381807712573cdd9de19a324094381a82bc618f9e2a7a28ef97479"}
Sep 11 03:58:00.632881 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.632858    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-k8wf2"]
Sep 11 03:58:00.637241 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.637218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8" event={"ID":"acd2b8a0-5f53-4cac-b56c-ea05f1896497","Type":"ContainerStarted","Data":"f0f96aad1947c38b3b98747172916c9b47974e87da52dcd7d97b9fda4eef5457"}
Sep 11 03:58:00.637348 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.637317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k8wf2"
Sep 11 03:58:00.639579 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.639563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-r9bmw\""
Sep 11 03:58:00.645883 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.645865    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k8wf2"]
Sep 11 03:58:00.714704 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.714674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgns\" (UniqueName: \"kubernetes.io/projected/fc6a5e9c-9a18-4c2f-ab94-f05fff027223-kube-api-access-kmgns\") pod \"authorino-59c974bd48-k8wf2\" (UID: \"fc6a5e9c-9a18-4c2f-ab94-f05fff027223\") " pod="kuadrant-system/authorino-59c974bd48-k8wf2"
Sep 11 03:58:00.815499 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.815472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgns\" (UniqueName: \"kubernetes.io/projected/fc6a5e9c-9a18-4c2f-ab94-f05fff027223-kube-api-access-kmgns\") pod \"authorino-59c974bd48-k8wf2\" (UID: \"fc6a5e9c-9a18-4c2f-ab94-f05fff027223\") " pod="kuadrant-system/authorino-59c974bd48-k8wf2"
Sep 11 03:58:00.822891 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.822836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgns\" (UniqueName: \"kubernetes.io/projected/fc6a5e9c-9a18-4c2f-ab94-f05fff027223-kube-api-access-kmgns\") pod \"authorino-59c974bd48-k8wf2\" (UID: \"fc6a5e9c-9a18-4c2f-ab94-f05fff027223\") " pod="kuadrant-system/authorino-59c974bd48-k8wf2"
Sep 11 03:58:00.946794 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:00.946762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k8wf2"
Sep 11 03:58:01.233495 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:01.233469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k8wf2"]
Sep 11 03:58:01.258797 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:58:01.258768    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc6a5e9c_9a18_4c2f_ab94_f05fff027223.slice/crio-c56f9811f6036ea110f79afbaeb5f175aaf2fa39db5a1ee9d840ff2066f54e93 WatchSource:0}: Error finding container c56f9811f6036ea110f79afbaeb5f175aaf2fa39db5a1ee9d840ff2066f54e93: Status 404 returned error can't find the container with id c56f9811f6036ea110f79afbaeb5f175aaf2fa39db5a1ee9d840ff2066f54e93
Sep 11 03:58:01.640041 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:01.639956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8" event={"ID":"acd2b8a0-5f53-4cac-b56c-ea05f1896497","Type":"ContainerStarted","Data":"bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749"}
Sep 11 03:58:01.640041 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:01.640026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:58:01.641399 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:01.641374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk" event={"ID":"84a96ab1-cd48-409d-90bc-7af13fbcea18","Type":"ContainerStarted","Data":"00c53acc437c7e5503311184ef4b29473f69cae9e6e3191f58e780f3c2dbb081"}
Sep 11 03:58:01.641522 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:01.641441    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:01.642521 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:01.642494    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k8wf2" event={"ID":"fc6a5e9c-9a18-4c2f-ab94-f05fff027223","Type":"ContainerStarted","Data":"c56f9811f6036ea110f79afbaeb5f175aaf2fa39db5a1ee9d840ff2066f54e93"}
Sep 11 03:58:01.656200 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:01.656154    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8" podStartSLOduration=1.50635617 podStartE2EDuration="2.656141666s" totalImagesPullingTime="1.149785496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:01.654332051 +0000 UTC m=+479.520351785" watchObservedRunningTime="2026-09-11 03:58:01.656141666 +0000 UTC m=+479.522161456"
Sep 11 03:58:01.669545 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:01.669504    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk" podStartSLOduration=1.907474109 podStartE2EDuration="2.669487531s" totalImagesPullingTime="762.013422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:01.66811256 +0000 UTC m=+479.534132294" watchObservedRunningTime="2026-09-11 03:58:01.669487531 +0000 UTC m=+479.535507266"
Sep 11 03:58:03.652707 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:03.652674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k8wf2" event={"ID":"fc6a5e9c-9a18-4c2f-ab94-f05fff027223","Type":"ContainerStarted","Data":"2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f"}
Sep 11 03:58:03.665619 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:03.665577    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-k8wf2" podStartSLOduration=2.152906721 podStartE2EDuration="3.66556152s" totalImagesPullingTime="1.512654799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:03.664309493 +0000 UTC m=+481.530329229" watchObservedRunningTime="2026-09-11 03:58:03.66556152 +0000 UTC m=+481.531581285"
Sep 11 03:58:12.648090 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:12.648059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-87jlk"
Sep 11 03:58:12.648513 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:12.648123    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:58:12.701713 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:12.701689    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfgx8"]
Sep 11 03:58:12.701865 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:12.701843    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8" podUID="acd2b8a0-5f53-4cac-b56c-ea05f1896497" containerName="limitador" containerID="cri-o://bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749" gracePeriod=30
Sep 11 03:58:13.238444 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.238419    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:58:13.308546 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.308472    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/acd2b8a0-5f53-4cac-b56c-ea05f1896497-config-file\") pod \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\" (UID: \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\") "
Sep 11 03:58:13.308546 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.308523    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh25w\" (UniqueName: \"kubernetes.io/projected/acd2b8a0-5f53-4cac-b56c-ea05f1896497-kube-api-access-hh25w\") pod \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\" (UID: \"acd2b8a0-5f53-4cac-b56c-ea05f1896497\") "
Sep 11 03:58:13.308833 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.308808    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd2b8a0-5f53-4cac-b56c-ea05f1896497-config-file" (OuterVolumeSpecName: "config-file") pod "acd2b8a0-5f53-4cac-b56c-ea05f1896497" (UID: "acd2b8a0-5f53-4cac-b56c-ea05f1896497"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 03:58:13.310522 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.310500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd2b8a0-5f53-4cac-b56c-ea05f1896497-kube-api-access-hh25w" (OuterVolumeSpecName: "kube-api-access-hh25w") pod "acd2b8a0-5f53-4cac-b56c-ea05f1896497" (UID: "acd2b8a0-5f53-4cac-b56c-ea05f1896497"). InnerVolumeSpecName "kube-api-access-hh25w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:13.409574 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.409553    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/acd2b8a0-5f53-4cac-b56c-ea05f1896497-config-file\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:58:13.409574 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.409573    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh25w\" (UniqueName: \"kubernetes.io/projected/acd2b8a0-5f53-4cac-b56c-ea05f1896497-kube-api-access-hh25w\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:58:13.693577 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.693511    2566 generic.go:356] "Generic (PLEG): container finished" podID="acd2b8a0-5f53-4cac-b56c-ea05f1896497" containerID="bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749" exitCode=0
Sep 11 03:58:13.693577 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.693552    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8"
Sep 11 03:58:13.693907 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.693605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8" event={"ID":"acd2b8a0-5f53-4cac-b56c-ea05f1896497","Type":"ContainerDied","Data":"bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749"}
Sep 11 03:58:13.693907 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.693651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wfgx8" event={"ID":"acd2b8a0-5f53-4cac-b56c-ea05f1896497","Type":"ContainerDied","Data":"f0f96aad1947c38b3b98747172916c9b47974e87da52dcd7d97b9fda4eef5457"}
Sep 11 03:58:13.693907 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.693672    2566 scope.go:117] "RemoveContainer" containerID="bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749"
Sep 11 03:58:13.702068 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.702048    2566 scope.go:117] "RemoveContainer" containerID="bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749"
Sep 11 03:58:13.702331 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:58:13.702314    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749\": container with ID starting with bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749 not found: ID does not exist" containerID="bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749"
Sep 11 03:58:13.702377 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.702341    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749"} err="failed to get container status \"bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749\": rpc error: code = NotFound desc = could not find container \"bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749\": container with ID starting with bec9c88d2bc38ef438d314017267d2737a084389b24a6a380677b7b8809be749 not found: ID does not exist"
Sep 11 03:58:13.713643 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.713621    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfgx8"]
Sep 11 03:58:13.720078 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:13.720059    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wfgx8"]
Sep 11 03:58:14.616567 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:14.616533    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acd2b8a0-5f53-4cac-b56c-ea05f1896497" path="/var/lib/kubelet/pods/acd2b8a0-5f53-4cac-b56c-ea05f1896497/volumes"
Sep 11 03:58:32.214338 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.214310    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k8wf2"]
Sep 11 03:58:32.214799 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.214534    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-k8wf2" podUID="fc6a5e9c-9a18-4c2f-ab94-f05fff027223" containerName="authorino" containerID="cri-o://2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f" gracePeriod=30
Sep 11 03:58:32.445751 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.445730    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k8wf2"
Sep 11 03:58:32.538252 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.538194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmgns\" (UniqueName: \"kubernetes.io/projected/fc6a5e9c-9a18-4c2f-ab94-f05fff027223-kube-api-access-kmgns\") pod \"fc6a5e9c-9a18-4c2f-ab94-f05fff027223\" (UID: \"fc6a5e9c-9a18-4c2f-ab94-f05fff027223\") "
Sep 11 03:58:32.540068 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.540045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6a5e9c-9a18-4c2f-ab94-f05fff027223-kube-api-access-kmgns" (OuterVolumeSpecName: "kube-api-access-kmgns") pod "fc6a5e9c-9a18-4c2f-ab94-f05fff027223" (UID: "fc6a5e9c-9a18-4c2f-ab94-f05fff027223"). InnerVolumeSpecName "kube-api-access-kmgns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:32.639451 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.639402    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmgns\" (UniqueName: \"kubernetes.io/projected/fc6a5e9c-9a18-4c2f-ab94-f05fff027223-kube-api-access-kmgns\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:58:32.765853 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.765827    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc6a5e9c-9a18-4c2f-ab94-f05fff027223" containerID="2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f" exitCode=0
Sep 11 03:58:32.765982 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.765876    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-k8wf2"
Sep 11 03:58:32.765982 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.765887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k8wf2" event={"ID":"fc6a5e9c-9a18-4c2f-ab94-f05fff027223","Type":"ContainerDied","Data":"2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f"}
Sep 11 03:58:32.765982 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.765927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-k8wf2" event={"ID":"fc6a5e9c-9a18-4c2f-ab94-f05fff027223","Type":"ContainerDied","Data":"c56f9811f6036ea110f79afbaeb5f175aaf2fa39db5a1ee9d840ff2066f54e93"}
Sep 11 03:58:32.765982 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.765943    2566 scope.go:117] "RemoveContainer" containerID="2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f"
Sep 11 03:58:32.773725 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.773708    2566 scope.go:117] "RemoveContainer" containerID="2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f"
Sep 11 03:58:32.774006 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:58:32.773988    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f\": container with ID starting with 2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f not found: ID does not exist" containerID="2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f"
Sep 11 03:58:32.774073 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.774013    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f"} err="failed to get container status \"2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f\": rpc error: code = NotFound desc = could not find container \"2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f\": container with ID starting with 2a08f4bb3cf7087b3417a661cbc245ea269a02ccba5cf35ace9484321d644f7f not found: ID does not exist"
Sep 11 03:58:32.786644 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.786621    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k8wf2"]
Sep 11 03:58:32.792495 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:32.792447    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-k8wf2"]
Sep 11 03:58:34.611368 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:34.611336    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc6a5e9c-9a18-4c2f-ab94-f05fff027223" path="/var/lib/kubelet/pods/fc6a5e9c-9a18-4c2f-ab94-f05fff027223/volumes"
Sep 11 03:58:36.850339 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.850310    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d74444df6-285gr"]
Sep 11 03:58:36.850784 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.850627    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc6a5e9c-9a18-4c2f-ab94-f05fff027223" containerName="authorino"
Sep 11 03:58:36.850784 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.850638    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc6a5e9c-9a18-4c2f-ab94-f05fff027223" containerName="authorino"
Sep 11 03:58:36.850784 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.850648    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acd2b8a0-5f53-4cac-b56c-ea05f1896497" containerName="limitador"
Sep 11 03:58:36.850784 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.850654    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="acd2b8a0-5f53-4cac-b56c-ea05f1896497" containerName="limitador"
Sep 11 03:58:36.850784 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.850716    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="acd2b8a0-5f53-4cac-b56c-ea05f1896497" containerName="limitador"
Sep 11 03:58:36.850784 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.850724    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc6a5e9c-9a18-4c2f-ab94-f05fff027223" containerName="authorino"
Sep 11 03:58:36.853235 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.853210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:36.855538 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.855508    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 03:58:36.855664 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.855550    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gp885\""
Sep 11 03:58:36.855664 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.855508    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 03:58:36.856504 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.856487    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 03:58:36.862014 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.861991    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d74444df6-285gr"]
Sep 11 03:58:36.967553 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.967524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:36.967689 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.967579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlt8\" (UniqueName: \"kubernetes.io/projected/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-kube-api-access-qrlt8\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:36.967689 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:36.967616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-webhook-cert\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.027814 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.027787    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-848cd9bd7-lnzhx"]
Sep 11 03:58:37.030086 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.030070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.050094 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.050067    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-lnzhx"]
Sep 11 03:58:37.068786 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.068764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.068873 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.068803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlt8\" (UniqueName: \"kubernetes.io/projected/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-kube-api-access-qrlt8\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.068873 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.068826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-webhook-cert\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.068957 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:58:37.068915    2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 03:58:37.068994 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:58:37.068979    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs podName:ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf nodeName:}" failed. No retries permitted until 2026-09-11 03:58:37.568957832 +0000 UTC m=+515.434977551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs") pod "maas-controller-5d74444df6-285gr" (UID: "ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf") : secret "maas-controller-metrics-tls" not found
Sep 11 03:58:37.071121 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.071102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-webhook-cert\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.082756 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.082737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlt8\" (UniqueName: \"kubernetes.io/projected/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-kube-api-access-qrlt8\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.169958 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.169912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.169958 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.169956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8hb\" (UniqueName: \"kubernetes.io/projected/bb8c3ca4-e38e-420e-8bc0-00d008dad517-kube-api-access-wv8hb\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.170081 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.169996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-webhook-cert\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.270749 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.270730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-webhook-cert\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.270828 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.270772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.270828 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.270797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8hb\" (UniqueName: \"kubernetes.io/projected/bb8c3ca4-e38e-420e-8bc0-00d008dad517-kube-api-access-wv8hb\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.270899 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:58:37.270882    2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 03:58:37.270951 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:58:37.270939    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs podName:bb8c3ca4-e38e-420e-8bc0-00d008dad517 nodeName:}" failed. No retries permitted until 2026-09-11 03:58:37.770921656 +0000 UTC m=+515.636941375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs") pod "maas-controller-848cd9bd7-lnzhx" (UID: "bb8c3ca4-e38e-420e-8bc0-00d008dad517") : secret "maas-controller-metrics-tls" not found
Sep 11 03:58:37.272732 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.272714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-webhook-cert\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.281777 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.281754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8hb\" (UniqueName: \"kubernetes.io/projected/bb8c3ca4-e38e-420e-8bc0-00d008dad517-kube-api-access-wv8hb\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.572758 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.572735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.574708 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.574691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs\") pod \"maas-controller-5d74444df6-285gr\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") " pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.764381 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.764358    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:37.774264 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.774239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.776432 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.776400    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs\") pod \"maas-controller-848cd9bd7-lnzhx\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") " pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:37.941557 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:37.941494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:38.028618 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:38.028594    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d74444df6-285gr"]
Sep 11 03:58:38.030747 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:58:38.030724    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5ee6e2_d49a_45ab_9a1b_c94e9c8763cf.slice/crio-8cc4e64a18d0dc9611eeef82167f9482f6c59026dc3557c6fa1ae1b91538e403 WatchSource:0}: Error finding container 8cc4e64a18d0dc9611eeef82167f9482f6c59026dc3557c6fa1ae1b91538e403: Status 404 returned error can't find the container with id 8cc4e64a18d0dc9611eeef82167f9482f6c59026dc3557c6fa1ae1b91538e403
Sep 11 03:58:38.208791 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:38.208773    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-lnzhx"]
Sep 11 03:58:38.210287 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:58:38.210264    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8c3ca4_e38e_420e_8bc0_00d008dad517.slice/crio-5aaaf73b5a8c5755fe22bb27467c249621953e8d6ce0b86f29e1e4872b2a572b WatchSource:0}: Error finding container 5aaaf73b5a8c5755fe22bb27467c249621953e8d6ce0b86f29e1e4872b2a572b: Status 404 returned error can't find the container with id 5aaaf73b5a8c5755fe22bb27467c249621953e8d6ce0b86f29e1e4872b2a572b
Sep 11 03:58:38.795239 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:38.795157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d74444df6-285gr" event={"ID":"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf","Type":"ContainerStarted","Data":"8cc4e64a18d0dc9611eeef82167f9482f6c59026dc3557c6fa1ae1b91538e403"}
Sep 11 03:58:38.797551 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:38.797520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-lnzhx" event={"ID":"bb8c3ca4-e38e-420e-8bc0-00d008dad517","Type":"ContainerStarted","Data":"5aaaf73b5a8c5755fe22bb27467c249621953e8d6ce0b86f29e1e4872b2a572b"}
Sep 11 03:58:40.810929 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:40.810887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-lnzhx" event={"ID":"bb8c3ca4-e38e-420e-8bc0-00d008dad517","Type":"ContainerStarted","Data":"ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28"}
Sep 11 03:58:40.811343 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:40.811010    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:40.812339 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:40.812312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d74444df6-285gr" event={"ID":"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf","Type":"ContainerStarted","Data":"1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf"}
Sep 11 03:58:40.812458 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:40.812369    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:40.831704 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:40.831624    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-848cd9bd7-lnzhx" podStartSLOduration=1.463273225 podStartE2EDuration="3.831609821s" totalImagesPullingTime="2.368336596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:40.830125289 +0000 UTC m=+518.696145023" watchObservedRunningTime="2026-09-11 03:58:40.831609821 +0000 UTC m=+518.697629574"
Sep 11 03:58:40.850177 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:40.850137    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d74444df6-285gr" podStartSLOduration=2.308241863 podStartE2EDuration="4.850124676s" totalImagesPullingTime="2.541882813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:58:40.848159876 +0000 UTC m=+518.714179611" watchObservedRunningTime="2026-09-11 03:58:40.850124676 +0000 UTC m=+518.716144410"
Sep 11 03:58:51.819777 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:51.819744    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 03:58:51.820253 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:51.819881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:51.884002 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:51.883969    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d74444df6-285gr"]
Sep 11 03:58:51.884219 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:51.884196    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d74444df6-285gr" podUID="ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" containerName="manager" containerID="cri-o://1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf" gracePeriod=30
Sep 11 03:58:52.117257 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.117234    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:52.283728 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.283695    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs\") pod \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") "
Sep 11 03:58:52.283874 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.283789    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrlt8\" (UniqueName: \"kubernetes.io/projected/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-kube-api-access-qrlt8\") pod \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") "
Sep 11 03:58:52.283874 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.283841    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-webhook-cert\") pod \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\" (UID: \"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf\") "
Sep 11 03:58:52.285769 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.285737    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" (UID: "ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:58:52.285870 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.285766    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" (UID: "ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 03:58:52.285870 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.285793    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-kube-api-access-qrlt8" (OuterVolumeSpecName: "kube-api-access-qrlt8") pod "ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" (UID: "ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf"). InnerVolumeSpecName "kube-api-access-qrlt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 03:58:52.384319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.384269    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-webhook-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:58:52.384319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.384287    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-metrics-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:58:52.384319 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.384296    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrlt8\" (UniqueName: \"kubernetes.io/projected/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf-kube-api-access-qrlt8\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 03:58:52.867235 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.867201    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" containerID="1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf" exitCode=0
Sep 11 03:58:52.867663 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.867249    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d74444df6-285gr"
Sep 11 03:58:52.867663 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.867258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d74444df6-285gr" event={"ID":"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf","Type":"ContainerDied","Data":"1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf"}
Sep 11 03:58:52.867663 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.867330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d74444df6-285gr" event={"ID":"ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf","Type":"ContainerDied","Data":"8cc4e64a18d0dc9611eeef82167f9482f6c59026dc3557c6fa1ae1b91538e403"}
Sep 11 03:58:52.867663 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.867352    2566 scope.go:117] "RemoveContainer" containerID="1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf"
Sep 11 03:58:52.875073 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.875060    2566 scope.go:117] "RemoveContainer" containerID="1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf"
Sep 11 03:58:52.875315 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:58:52.875298    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf\": container with ID starting with 1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf not found: ID does not exist" containerID="1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf"
Sep 11 03:58:52.875369 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.875322    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf"} err="failed to get container status \"1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf\": rpc error: code = NotFound desc = could not find container \"1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf\": container with ID starting with 1ae041a63f1fe420926e6577360ebe4f395be719c71c410a719e794cab58d4cf not found: ID does not exist"
Sep 11 03:58:52.882168 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.882149    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d74444df6-285gr"]
Sep 11 03:58:52.885957 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:52.885939    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d74444df6-285gr"]
Sep 11 03:58:54.611318 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:58:54.611287    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" path="/var/lib/kubelet/pods/ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf/volumes"
Sep 11 03:59:13.984152 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.984117    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"]
Sep 11 03:59:13.984586 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.984568    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" containerName="manager"
Sep 11 03:59:13.984586 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.984582    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" containerName="manager"
Sep 11 03:59:13.984690 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.984678    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5ee6e2-d49a-45ab-9a1b-c94e9c8763cf" containerName="manager"
Sep 11 03:59:13.992997 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.992976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:13.995944 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.995912    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 03:59:13.995944 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.995913    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 03:59:13.996096 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.995916    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 03:59:13.996720 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.996693    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wb8bv\""
Sep 11 03:59:13.996845 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.996697    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 03:59:13.997927 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:13.997908    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"]
Sep 11 03:59:14.140552 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.140526    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rq9\" (UniqueName: \"kubernetes.io/projected/79b25eae-b255-4c81-8431-dc1cfa4b269c-kube-api-access-x6rq9\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.140687 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.140560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25eae-b255-4c81-8431-dc1cfa4b269c-openshift-service-ca\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.140687 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.140576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-metrics-certs\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.140687 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.140605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.241194 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.241135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rq9\" (UniqueName: \"kubernetes.io/projected/79b25eae-b255-4c81-8431-dc1cfa4b269c-kube-api-access-x6rq9\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.241194 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.241164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25eae-b255-4c81-8431-dc1cfa4b269c-openshift-service-ca\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.241194 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.241182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-metrics-certs\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.241434 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.241197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.241434 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:59:14.241284    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 11 03:59:14.241434 ip-10-0-131-122 kubenswrapper[2566]: E0911 03:59:14.241340    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls podName:79b25eae-b255-4c81-8431-dc1cfa4b269c nodeName:}" failed. No retries permitted until 2026-09-11 03:59:14.74131937 +0000 UTC m=+552.607339083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls") pod "maas-api-cfc75856f-h8kk4" (UID: "79b25eae-b255-4c81-8431-dc1cfa4b269c") : secret "maas-api-serving-cert" not found
Sep 11 03:59:14.241782 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.241762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25eae-b255-4c81-8431-dc1cfa4b269c-openshift-service-ca\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.243466 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.243439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-metrics-certs\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.250883 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.250855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rq9\" (UniqueName: \"kubernetes.io/projected/79b25eae-b255-4c81-8431-dc1cfa4b269c-kube-api-access-x6rq9\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.495149 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.495074    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"]
Sep 11 03:59:14.498952 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.498934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:14.501478 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.501458    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-t52vh\""
Sep 11 03:59:14.501589 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.501506    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 03:59:14.513440 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.513419    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"]
Sep 11 03:59:14.605575 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.605547    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 03:59:14.643905 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.643875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rh56\" (UniqueName: \"kubernetes.io/projected/3338e8c2-2273-4099-aed2-4a55c00c007e-kube-api-access-2rh56\") pod \"payload-processing-5b54b7fbb7-l5kqq\" (UID: \"3338e8c2-2273-4099-aed2-4a55c00c007e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:14.644039 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.643952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3338e8c2-2273-4099-aed2-4a55c00c007e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-l5kqq\" (UID: \"3338e8c2-2273-4099-aed2-4a55c00c007e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:14.744826 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.744793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.744986 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.744832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3338e8c2-2273-4099-aed2-4a55c00c007e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-l5kqq\" (UID: \"3338e8c2-2273-4099-aed2-4a55c00c007e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:14.745046 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.745005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rh56\" (UniqueName: \"kubernetes.io/projected/3338e8c2-2273-4099-aed2-4a55c00c007e-kube-api-access-2rh56\") pod \"payload-processing-5b54b7fbb7-l5kqq\" (UID: \"3338e8c2-2273-4099-aed2-4a55c00c007e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:14.745770 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.745711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3338e8c2-2273-4099-aed2-4a55c00c007e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-l5kqq\" (UID: \"3338e8c2-2273-4099-aed2-4a55c00c007e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:14.747594 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.747571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls\") pod \"maas-api-cfc75856f-h8kk4\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") " pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:14.759122 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.759103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rh56\" (UniqueName: \"kubernetes.io/projected/3338e8c2-2273-4099-aed2-4a55c00c007e-kube-api-access-2rh56\") pod \"payload-processing-5b54b7fbb7-l5kqq\" (UID: \"3338e8c2-2273-4099-aed2-4a55c00c007e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:14.808953 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.808927    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:14.905765 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:14.905732    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:15.105021 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:15.104937    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"]
Sep 11 03:59:15.108750 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:59:15.108726    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3338e8c2_2273_4099_aed2_4a55c00c007e.slice/crio-1566524bbc89fcb70fc5783b2c9a4563dc09613a23b42de382fdd05aebebad0e WatchSource:0}: Error finding container 1566524bbc89fcb70fc5783b2c9a4563dc09613a23b42de382fdd05aebebad0e: Status 404 returned error can't find the container with id 1566524bbc89fcb70fc5783b2c9a4563dc09613a23b42de382fdd05aebebad0e
Sep 11 03:59:15.175273 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:15.175255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"]
Sep 11 03:59:15.176707 ip-10-0-131-122 kubenswrapper[2566]: W0911 03:59:15.176687    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b25eae_b255_4c81_8431_dc1cfa4b269c.slice/crio-be17bcb824d71d1917fe0ac8350a7f3b7cf527229e080c718d78f96bcabb4ea3 WatchSource:0}: Error finding container be17bcb824d71d1917fe0ac8350a7f3b7cf527229e080c718d78f96bcabb4ea3: Status 404 returned error can't find the container with id be17bcb824d71d1917fe0ac8350a7f3b7cf527229e080c718d78f96bcabb4ea3
Sep 11 03:59:15.975180 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:15.975143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4" event={"ID":"79b25eae-b255-4c81-8431-dc1cfa4b269c","Type":"ContainerStarted","Data":"be17bcb824d71d1917fe0ac8350a7f3b7cf527229e080c718d78f96bcabb4ea3"}
Sep 11 03:59:15.976700 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:15.976669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq" event={"ID":"3338e8c2-2273-4099-aed2-4a55c00c007e","Type":"ContainerStarted","Data":"1566524bbc89fcb70fc5783b2c9a4563dc09613a23b42de382fdd05aebebad0e"}
Sep 11 03:59:18.993321 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:18.993049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq" event={"ID":"3338e8c2-2273-4099-aed2-4a55c00c007e","Type":"ContainerStarted","Data":"849dc42932c98ffd66c55b3a05ad27d977fd7b76addb2906d7130c9969b515a2"}
Sep 11 03:59:18.993321 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:18.993231    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:18.994775 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:18.994746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4" event={"ID":"79b25eae-b255-4c81-8431-dc1cfa4b269c","Type":"ContainerStarted","Data":"0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7"}
Sep 11 03:59:18.994885 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:18.994809    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:19.014075 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:19.014033    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq" podStartSLOduration=1.707327965 podStartE2EDuration="5.014019798s" totalImagesPullingTime="3.306691833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:19.012314648 +0000 UTC m=+556.878334383" watchObservedRunningTime="2026-09-11 03:59:19.014019798 +0000 UTC m=+556.880039533"
Sep 11 03:59:19.029031 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:19.028992    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4" podStartSLOduration=2.7897256820000003 podStartE2EDuration="6.028974886s" totalImagesPullingTime="3.239249204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:19.028357654 +0000 UTC m=+556.894377388" watchObservedRunningTime="2026-09-11 03:59:19.028974886 +0000 UTC m=+556.894994640"
Sep 11 03:59:24.018383 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:24.018355    2566 generic.go:356] "Generic (PLEG): container finished" podID="2028fff4-7d56-4cee-be2c-0e1e054793a5" containerID="6ff40a712c54f71285c70c7c99c104660be8d34c330c88be5e87a4fc084e829d" exitCode=0
Sep 11 03:59:24.018738 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:24.018428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" event={"ID":"2028fff4-7d56-4cee-be2c-0e1e054793a5","Type":"ContainerDied","Data":"6ff40a712c54f71285c70c7c99c104660be8d34c330c88be5e87a4fc084e829d"}
Sep 11 03:59:24.018781 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:24.018741    2566 scope.go:117] "RemoveContainer" containerID="6ff40a712c54f71285c70c7c99c104660be8d34c330c88be5e87a4fc084e829d"
Sep 11 03:59:24.019577 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:24.019554    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:59:24.019640 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:24.019612    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:59:24.019676 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:24.019639    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 03:59:25.005525 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:25.005490    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 03:59:25.024493 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:25.024461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" event={"ID":"2028fff4-7d56-4cee-be2c-0e1e054793a5","Type":"ContainerStarted","Data":"af7a28c107604a170a856b596d06d875f27ec3cfb3181073d30fe1f077a03c58"}
Sep 11 03:59:25.490811 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:25.490776    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:59:25.491926 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:25.491903    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nmjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 03:59:25.492025 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:25.491958    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" podUID="2028fff4-7d56-4cee-be2c-0e1e054793a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 03:59:26.491265 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:26.491225    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nmjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 03:59:26.491658 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:26.491279    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" podUID="2028fff4-7d56-4cee-be2c-0e1e054793a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 03:59:27.491623 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:27.491585    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nmjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 03:59:27.492013 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:27.491645    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" podUID="2028fff4-7d56-4cee-be2c-0e1e054793a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 03:59:28.491168 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:28.491129    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nmjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 03:59:28.491333 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:28.491185    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" podUID="2028fff4-7d56-4cee-be2c-0e1e054793a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 03:59:29.490674 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:29.490643    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nmjcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 11 03:59:29.491069 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:29.490695    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb" podUID="2028fff4-7d56-4cee-be2c-0e1e054793a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 11 03:59:29.627510 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:29.627472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:59:30.001970 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:30.001944    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-l5kqq"
Sep 11 03:59:30.494495 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:30.494467    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:59:30.495305 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:30.495287    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nmjcb"
Sep 11 03:59:47.290956 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:47.290921    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 03:59:59.109885 ip-10-0-131-122 kubenswrapper[2566]: I0911 03:59:59.109855    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:00:02.585441 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:02.585399    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:00:02.590936 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:02.590916    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:00:04.092207 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:04.092174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:00:14.492108 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.492075    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"]
Sep 11 04:00:14.495092 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.495076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.497757 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.497735    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jkjzv\""
Sep 11 04:00:14.498715 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.498698    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 04:00:14.498816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.498738    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 04:00:14.498816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.498783    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 04:00:14.508302 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.508249    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"]
Sep 11 04:00:14.530740 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.530715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.530862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.530753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.530862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.530776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsm7\" (UniqueName: \"kubernetes.io/projected/e555900e-7f9f-4d72-9d6d-abe296285ccf-kube-api-access-wgsm7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.530862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.530793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.530862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.530845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.531037 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.530897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e555900e-7f9f-4d72-9d6d-abe296285ccf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.631676 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.631644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.631862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.631704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsm7\" (UniqueName: \"kubernetes.io/projected/e555900e-7f9f-4d72-9d6d-abe296285ccf-kube-api-access-wgsm7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.631862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.631738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.631862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.631764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.631862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.631795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e555900e-7f9f-4d72-9d6d-abe296285ccf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.632052 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.631902    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.632185 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.632162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.632388 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.632364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.632533 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.632516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.634250 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.634227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e555900e-7f9f-4d72-9d6d-abe296285ccf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.634318 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.634300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e555900e-7f9f-4d72-9d6d-abe296285ccf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.639837 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.639815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsm7\" (UniqueName: \"kubernetes.io/projected/e555900e-7f9f-4d72-9d6d-abe296285ccf-kube-api-access-wgsm7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nw4r6\" (UID: \"e555900e-7f9f-4d72-9d6d-abe296285ccf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:14.806317 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:14.806249    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:15.087775 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:15.087745    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"]
Sep 11 04:00:15.088481 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:00:15.088460    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode555900e_7f9f_4d72_9d6d_abe296285ccf.slice/crio-dd3cf10e159886d9e38013645a1fbf4e51f362f768e4157a3c201f5df9fbfb2b WatchSource:0}: Error finding container dd3cf10e159886d9e38013645a1fbf4e51f362f768e4157a3c201f5df9fbfb2b: Status 404 returned error can't find the container with id dd3cf10e159886d9e38013645a1fbf4e51f362f768e4157a3c201f5df9fbfb2b
Sep 11 04:00:15.089826 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:15.089807    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:00:15.231690 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:15.231655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6" event={"ID":"e555900e-7f9f-4d72-9d6d-abe296285ccf","Type":"ContainerStarted","Data":"dd3cf10e159886d9e38013645a1fbf4e51f362f768e4157a3c201f5df9fbfb2b"}
Sep 11 04:00:15.696975 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:15.696941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:00:28.888873 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:28.888784    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:00:30.299756 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:30.299663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6" event={"ID":"e555900e-7f9f-4d72-9d6d-abe296285ccf","Type":"ContainerStarted","Data":"cf4dc18fd83d31e2182483b009f301603d4ba319526aa4c763c00eafdda1c68b"}
Sep 11 04:00:30.317337 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:30.317292    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6" podStartSLOduration=1.409740456 podStartE2EDuration="16.317276924s" totalImagesPullingTime="14.907536468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:30.316295898 +0000 UTC m=+628.182315625" watchObservedRunningTime="2026-09-11 04:00:30.317276924 +0000 UTC m=+628.183296701"
Sep 11 04:00:31.756900 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.756868    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"]
Sep 11 04:00:31.766326 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.760457    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.776878 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.776855    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 04:00:31.781609 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.781588    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"]
Sep 11 04:00:31.862316 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.862279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2158593-b29c-496d-a333-01027d5f3957-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.862316 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.862313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.862517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.862343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.862517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.862398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.862517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.862474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbdn\" (UniqueName: \"kubernetes.io/projected/d2158593-b29c-496d-a333-01027d5f3957-kube-api-access-8sbdn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.862649 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.862534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.963731 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.963697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.963731 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.963736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2158593-b29c-496d-a333-01027d5f3957-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.963952 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.963753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.963952 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.963777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.963952 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.963792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.963952 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.963934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbdn\" (UniqueName: \"kubernetes.io/projected/d2158593-b29c-496d-a333-01027d5f3957-kube-api-access-8sbdn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.964158 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.964136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.964220 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.964201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.964285 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.964251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.966067 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.966046    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2158593-b29c-496d-a333-01027d5f3957-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.966198 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.966181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2158593-b29c-496d-a333-01027d5f3957-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:31.972204 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:31.972177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbdn\" (UniqueName: \"kubernetes.io/projected/d2158593-b29c-496d-a333-01027d5f3957-kube-api-access-8sbdn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctbnw\" (UID: \"d2158593-b29c-496d-a333-01027d5f3957\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:32.074699 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:32.074639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:32.344132 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:32.344105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"]
Sep 11 04:00:32.346503 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:00:32.346470    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2158593_b29c_496d_a333_01027d5f3957.slice/crio-f750c08bbaae71e2083ee05851bf129fb139ad0fbb5d83cb135ff62c864ed206 WatchSource:0}: Error finding container f750c08bbaae71e2083ee05851bf129fb139ad0fbb5d83cb135ff62c864ed206: Status 404 returned error can't find the container with id f750c08bbaae71e2083ee05851bf129fb139ad0fbb5d83cb135ff62c864ed206
Sep 11 04:00:33.313575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.313538    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw" event={"ID":"d2158593-b29c-496d-a333-01027d5f3957","Type":"ContainerStarted","Data":"43ffe142f2615ebb4148ec7bc68155a6b1016169cae6e87b3946bc558de8ef65"}
Sep 11 04:00:33.314050 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.313601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw" event={"ID":"d2158593-b29c-496d-a333-01027d5f3957","Type":"ContainerStarted","Data":"f750c08bbaae71e2083ee05851bf129fb139ad0fbb5d83cb135ff62c864ed206"}
Sep 11 04:00:33.332564 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.332518    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw" podStartSLOduration=2.145818921 podStartE2EDuration="2.332502637s" totalImagesPullingTime="186.683716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:33.331291814 +0000 UTC m=+631.197311559" watchObservedRunningTime="2026-09-11 04:00:33.332502637 +0000 UTC m=+631.198522420"
Sep 11 04:00:33.395383 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.395354    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"]
Sep 11 04:00:33.398747 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.398729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.401087 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.401065    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 04:00:33.408852 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.408829    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"]
Sep 11 04:00:33.475575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.475547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.475687 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.475584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.475687 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.475629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e235ecec-31f0-4974-add9-7bb428973b8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.475767 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.475701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.475767 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.475734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.475767 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.475756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc49x\" (UniqueName: \"kubernetes.io/projected/e235ecec-31f0-4974-add9-7bb428973b8c-kube-api-access-wc49x\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc49x\" (UniqueName: \"kubernetes.io/projected/e235ecec-31f0-4974-add9-7bb428973b8c-kube-api-access-wc49x\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576590 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576590 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576590 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e235ecec-31f0-4974-add9-7bb428973b8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576763 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576763 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576749    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.576941 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.576908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.579246 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.579221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e235ecec-31f0-4974-add9-7bb428973b8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.579389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.579373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e235ecec-31f0-4974-add9-7bb428973b8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.585298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.585276    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc49x\" (UniqueName: \"kubernetes.io/projected/e235ecec-31f0-4974-add9-7bb428973b8c-kube-api-access-wc49x\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2\" (UID: \"e235ecec-31f0-4974-add9-7bb428973b8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.709387 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.709355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:33.978452 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:33.978428    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"]
Sep 11 04:00:33.980881 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:00:33.980855    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode235ecec_31f0_4974_add9_7bb428973b8c.slice/crio-e1fcf647039f7ef99485a7872b70768d471e05231af935ca554b57bd9abf0ef0 WatchSource:0}: Error finding container e1fcf647039f7ef99485a7872b70768d471e05231af935ca554b57bd9abf0ef0: Status 404 returned error can't find the container with id e1fcf647039f7ef99485a7872b70768d471e05231af935ca554b57bd9abf0ef0
Sep 11 04:00:34.320672 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:34.320645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2" event={"ID":"e235ecec-31f0-4974-add9-7bb428973b8c","Type":"ContainerStarted","Data":"97acfd7bd39b02e8c66f8faaed97de2bc18c9fd35b430cc4ea33e3798b2d3614"}
Sep 11 04:00:34.321014 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:34.320685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2" event={"ID":"e235ecec-31f0-4974-add9-7bb428973b8c","Type":"ContainerStarted","Data":"e1fcf647039f7ef99485a7872b70768d471e05231af935ca554b57bd9abf0ef0"}
Sep 11 04:00:34.338465 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:34.338426    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2" podStartSLOduration=1.055047559 podStartE2EDuration="1.338393725s" totalImagesPullingTime="283.346166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:34.336324081 +0000 UTC m=+632.202343817" watchObservedRunningTime="2026-09-11 04:00:34.338393725 +0000 UTC m=+632.204413460"
Sep 11 04:00:34.711994 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:34.711963    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:00:34.806641 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:34.806608    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:34.806797 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:34.806670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:34.818245 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:34.818221    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:35.336845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:35.336818    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nw4r6"
Sep 11 04:00:42.074957 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:42.074926    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:42.075331 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:42.075079    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:42.087046 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:42.087029    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:42.374523 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:42.374455    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctbnw"
Sep 11 04:00:43.710167 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:43.710122    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:43.710584 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:43.710196    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:43.722253 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:43.722233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:00:44.382660 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:00:44.382633    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2"
Sep 11 04:01:47.548051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:47.548018    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-600aef"]
Sep 11 04:01:47.551461 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:47.551445    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:47.554785 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:47.554767    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mhkpw\""
Sep 11 04:01:47.565157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:47.565137    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-600aef"]
Sep 11 04:01:47.679617 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:47.679593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klc5h\" (UniqueName: \"kubernetes.io/projected/1da2ffe3-253f-4a71-a33d-09ee88c16ad2-kube-api-access-klc5h\") pod \"test-curl-7498-600aef\" (UID: \"1da2ffe3-253f-4a71-a33d-09ee88c16ad2\") " pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:47.780343 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:47.780314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klc5h\" (UniqueName: \"kubernetes.io/projected/1da2ffe3-253f-4a71-a33d-09ee88c16ad2-kube-api-access-klc5h\") pod \"test-curl-7498-600aef\" (UID: \"1da2ffe3-253f-4a71-a33d-09ee88c16ad2\") " pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:47.788485 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:47.788456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klc5h\" (UniqueName: \"kubernetes.io/projected/1da2ffe3-253f-4a71-a33d-09ee88c16ad2-kube-api-access-klc5h\") pod \"test-curl-7498-600aef\" (UID: \"1da2ffe3-253f-4a71-a33d-09ee88c16ad2\") " pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:47.860517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:47.860452    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:48.127826 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:48.127803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-600aef"]
Sep 11 04:01:48.130606 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:01:48.130579    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1da2ffe3_253f_4a71_a33d_09ee88c16ad2.slice/crio-ef17a72a3f90161416be9277c8753adb07ec2d9d763e9bc5aa9d46dd95b36e56 WatchSource:0}: Error finding container ef17a72a3f90161416be9277c8753adb07ec2d9d763e9bc5aa9d46dd95b36e56: Status 404 returned error can't find the container with id ef17a72a3f90161416be9277c8753adb07ec2d9d763e9bc5aa9d46dd95b36e56
Sep 11 04:01:48.659421 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:48.659383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-600aef" event={"ID":"1da2ffe3-253f-4a71-a33d-09ee88c16ad2","Type":"ContainerStarted","Data":"ef17a72a3f90161416be9277c8753adb07ec2d9d763e9bc5aa9d46dd95b36e56"}
Sep 11 04:01:49.305964 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:49.305929    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:01:50.670522 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:50.670392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-600aef" event={"ID":"1da2ffe3-253f-4a71-a33d-09ee88c16ad2","Type":"ContainerStarted","Data":"48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc"}
Sep 11 04:01:50.728881 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:50.728830    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7498-600aef" podStartSLOduration=1.566603652 podStartE2EDuration="3.728813452s" totalImagesPullingTime="2.1622098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:50.727595123 +0000 UTC m=+708.593614856" watchObservedRunningTime="2026-09-11 04:01:50.728813452 +0000 UTC m=+708.594833186"
Sep 11 04:01:51.108814 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.108778    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"]
Sep 11 04:01:51.112052 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.112037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:51.114455 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.114435    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-00a938\""
Sep 11 04:01:51.123277 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.123254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"]
Sep 11 04:01:51.211811 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.211781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85gg\" (UniqueName: \"kubernetes.io/projected/d820ebb6-2774-4e2f-9858-f03f1974d2f2-kube-api-access-w85gg\") pod \"payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t\" (UID: \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:51.211937 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.211826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d820ebb6-2774-4e2f-9858-f03f1974d2f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t\" (UID: \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:51.312913 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.312879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85gg\" (UniqueName: \"kubernetes.io/projected/d820ebb6-2774-4e2f-9858-f03f1974d2f2-kube-api-access-w85gg\") pod \"payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t\" (UID: \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:51.313068 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.313023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d820ebb6-2774-4e2f-9858-f03f1974d2f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t\" (UID: \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:51.313676 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.313659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d820ebb6-2774-4e2f-9858-f03f1974d2f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t\" (UID: \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:51.321696 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.321674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85gg\" (UniqueName: \"kubernetes.io/projected/d820ebb6-2774-4e2f-9858-f03f1974d2f2-kube-api-access-w85gg\") pod \"payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t\" (UID: \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:51.340221 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.340194    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-600aef"]
Sep 11 04:01:51.345433 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.345392    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-600aef"]
Sep 11 04:01:51.421313 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.421237    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:51.455857 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.455822    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-e78f30"]
Sep 11 04:01:51.456175 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.456160    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da2ffe3-253f-4a71-a33d-09ee88c16ad2" containerName="test-curl-7498-600aef"
Sep 11 04:01:51.456175 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.456174    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da2ffe3-253f-4a71-a33d-09ee88c16ad2" containerName="test-curl-7498-600aef"
Sep 11 04:01:51.456308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.456248    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da2ffe3-253f-4a71-a33d-09ee88c16ad2" containerName="test-curl-7498-600aef"
Sep 11 04:01:51.461539 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.459650    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:51.467811 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.467789    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-e78f30"]
Sep 11 04:01:51.615822 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.615790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf5lr\" (UniqueName: \"kubernetes.io/projected/56e3e3da-c795-424a-88bc-5ff2940cc816-kube-api-access-sf5lr\") pod \"test-curl-7498-e78f30\" (UID: \"56e3e3da-c795-424a-88bc-5ff2940cc816\") " pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:51.675283 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:51.675208    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7498-600aef" container="test-curl-7498-600aef"
Sep 11 04:01:51.694493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.694474    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"]
Sep 11 04:01:51.696334 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:01:51.696309    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd820ebb6_2774_4e2f_9858_f03f1974d2f2.slice/crio-a44c061f345c01bd40aafd4a1aea67c4d7dab5d112fdfc73e3beb66f6c7d8f94 WatchSource:0}: Error finding container a44c061f345c01bd40aafd4a1aea67c4d7dab5d112fdfc73e3beb66f6c7d8f94: Status 404 returned error can't find the container with id a44c061f345c01bd40aafd4a1aea67c4d7dab5d112fdfc73e3beb66f6c7d8f94
Sep 11 04:01:51.716792 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.716772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf5lr\" (UniqueName: \"kubernetes.io/projected/56e3e3da-c795-424a-88bc-5ff2940cc816-kube-api-access-sf5lr\") pod \"test-curl-7498-e78f30\" (UID: \"56e3e3da-c795-424a-88bc-5ff2940cc816\") " pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:51.729193 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.729169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf5lr\" (UniqueName: \"kubernetes.io/projected/56e3e3da-c795-424a-88bc-5ff2940cc816-kube-api-access-sf5lr\") pod \"test-curl-7498-e78f30\" (UID: \"56e3e3da-c795-424a-88bc-5ff2940cc816\") " pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:51.776324 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:51.776300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:52.048707 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.048683    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-e78f30"]
Sep 11 04:01:52.053602 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:01:52.053545    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56e3e3da_c795_424a_88bc_5ff2940cc816.slice/crio-df5b4f549652a6ed869be68d3cae5032917c3392e0a4bd0987f9beb26bd8e833 WatchSource:0}: Error finding container df5b4f549652a6ed869be68d3cae5032917c3392e0a4bd0987f9beb26bd8e833: Status 404 returned error can't find the container with id df5b4f549652a6ed869be68d3cae5032917c3392e0a4bd0987f9beb26bd8e833
Sep 11 04:01:52.681171 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.681137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-e78f30" event={"ID":"56e3e3da-c795-424a-88bc-5ff2940cc816","Type":"ContainerStarted","Data":"b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a"}
Sep 11 04:01:52.681645 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.681182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-e78f30" event={"ID":"56e3e3da-c795-424a-88bc-5ff2940cc816","Type":"ContainerStarted","Data":"df5b4f549652a6ed869be68d3cae5032917c3392e0a4bd0987f9beb26bd8e833"}
Sep 11 04:01:52.682808 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.682779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t" event={"ID":"d820ebb6-2774-4e2f-9858-f03f1974d2f2","Type":"ContainerStarted","Data":"8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189"}
Sep 11 04:01:52.682924 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.682830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t" event={"ID":"d820ebb6-2774-4e2f-9858-f03f1974d2f2","Type":"ContainerStarted","Data":"a44c061f345c01bd40aafd4a1aea67c4d7dab5d112fdfc73e3beb66f6c7d8f94"}
Sep 11 04:01:52.682924 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.682860    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:01:52.682924 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.682886    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7498-600aef" podUID="1da2ffe3-253f-4a71-a33d-09ee88c16ad2" containerName="test-curl-7498-600aef" containerID="cri-o://48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc" gracePeriod=2
Sep 11 04:01:52.695824 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.695781    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7498-e78f30" podStartSLOduration=1.695768144 podStartE2EDuration="1.695768144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:52.695048345 +0000 UTC m=+710.561068092" watchObservedRunningTime="2026-09-11 04:01:52.695768144 +0000 UTC m=+710.561787878"
Sep 11 04:01:52.715420 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:52.715370    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t" podStartSLOduration=1.715358301 podStartE2EDuration="1.715358301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:52.714089353 +0000 UTC m=+710.580109087" watchObservedRunningTime="2026-09-11 04:01:52.715358301 +0000 UTC m=+710.581378035"
Sep 11 04:01:53.250119 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:53.250083    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-e78f30"]
Sep 11 04:01:53.253828 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:53.253804    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-e78f30"]
Sep 11 04:01:53.686882 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:53.686854    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7498-e78f30" container="test-curl-7498-e78f30"
Sep 11 04:01:54.026901 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.026876    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"]
Sep 11 04:01:54.027245 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.027233    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e3e3da-c795-424a-88bc-5ff2940cc816" containerName="test-curl-7498-e78f30"
Sep 11 04:01:54.027287 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.027248    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e3e3da-c795-424a-88bc-5ff2940cc816" containerName="test-curl-7498-e78f30"
Sep 11 04:01:54.027319 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.027307    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e3e3da-c795-424a-88bc-5ff2940cc816" containerName="test-curl-7498-e78f30"
Sep 11 04:01:54.030654 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.030630    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.033724 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.033703    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-944e93-openshift-default-dockercfg-pt6p2\""
Sep 11 04:01:54.045565 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.045540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"]
Sep 11 04:01:54.140578 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-certs\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.140680 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-podinfo\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.140680 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-token\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.140680 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxsz4\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-kube-api-access-mxsz4\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.140680 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-credential-socket\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.140816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-socket\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.140816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-envoy\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.140816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6208c03-d739-42c9-a0e0-2521d4907b94-istiod-ca-cert\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.140816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.140793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-data\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.241885 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.241859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6208c03-d739-42c9-a0e0-2521d4907b94-istiod-ca-cert\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242030 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.241907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-data\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242030 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.241937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-certs\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242030 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.241961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-podinfo\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242030 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.241995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-token\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242030 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.242018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxsz4\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-kube-api-access-mxsz4\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242280 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.242080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-credential-socket\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242280 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.242110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-socket\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242280 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.242137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-envoy\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242858 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.242730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6208c03-d739-42c9-a0e0-2521d4907b94-istiod-ca-cert\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.242992 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.242928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-socket\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.243198 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.243173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-credential-socket\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.243512 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.243402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-certs\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.243771 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.243743    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-data\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.245195 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.245140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-envoy\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.245288 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.245271    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-podinfo\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.252081 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.252055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-token\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.252652 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.252622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxsz4\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-kube-api-access-mxsz4\") pod \"e2e-autores-944e93-openshift-default-777cdf84fd-52k6f\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") " pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.341466 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.341396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:54.628178 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.628153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"]
Sep 11 04:01:54.629764 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:01:54.629737    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6208c03_d739_42c9_a0e0_2521d4907b94.slice/crio-97bbcdaee37d27b2080da871fb43a4f9834f47831dec7cd2bfb2421ef09a24c2 WatchSource:0}: Error finding container 97bbcdaee37d27b2080da871fb43a4f9834f47831dec7cd2bfb2421ef09a24c2: Status 404 returned error can't find the container with id 97bbcdaee37d27b2080da871fb43a4f9834f47831dec7cd2bfb2421ef09a24c2
Sep 11 04:01:54.631922 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.631892    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:01:54.632025 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.631954    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:01:54.632025 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.631980    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:01:54.693879 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.693855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" event={"ID":"b6208c03-d739-42c9-a0e0-2521d4907b94","Type":"ContainerStarted","Data":"97bbcdaee37d27b2080da871fb43a4f9834f47831dec7cd2bfb2421ef09a24c2"}
Sep 11 04:01:54.694225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.693891    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7498-e78f30" podUID="56e3e3da-c795-424a-88bc-5ff2940cc816" containerName="test-curl-7498-e78f30" containerID="cri-o://b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a" gracePeriod=2
Sep 11 04:01:54.873202 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:54.873180    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:55.049557 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.049481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klc5h\" (UniqueName: \"kubernetes.io/projected/1da2ffe3-253f-4a71-a33d-09ee88c16ad2-kube-api-access-klc5h\") pod \"1da2ffe3-253f-4a71-a33d-09ee88c16ad2\" (UID: \"1da2ffe3-253f-4a71-a33d-09ee88c16ad2\") "
Sep 11 04:01:55.051399 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.051377    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da2ffe3-253f-4a71-a33d-09ee88c16ad2-kube-api-access-klc5h" (OuterVolumeSpecName: "kube-api-access-klc5h") pod "1da2ffe3-253f-4a71-a33d-09ee88c16ad2" (UID: "1da2ffe3-253f-4a71-a33d-09ee88c16ad2"). InnerVolumeSpecName "kube-api-access-klc5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:01:55.150513 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.150483    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klc5h\" (UniqueName: \"kubernetes.io/projected/1da2ffe3-253f-4a71-a33d-09ee88c16ad2-kube-api-access-klc5h\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:01:55.705360 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.705321    2566 generic.go:356] "Generic (PLEG): container finished" podID="1da2ffe3-253f-4a71-a33d-09ee88c16ad2" containerID="48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc" exitCode=137
Sep 11 04:01:55.705852 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.705462    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:55.705925 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.705503    2566 scope.go:117] "RemoveContainer" containerID="48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc"
Sep 11 04:01:55.708018 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.707617    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" event={"ID":"b6208c03-d739-42c9-a0e0-2521d4907b94","Type":"ContainerStarted","Data":"32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329"}
Sep 11 04:01:55.715831 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.715817    2566 scope.go:117] "RemoveContainer" containerID="48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc"
Sep 11 04:01:55.716066 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:55.716047    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc\": container with ID starting with 48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc not found: ID does not exist" containerID="48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc"
Sep 11 04:01:55.716131 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.716076    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc"} err="failed to get container status \"48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc\": rpc error: code = NotFound desc = could not find container \"48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc\": container with ID starting with 48fa0d24ee997eac7c67b45b8d0db90f7fc0f1203b71e34eff4dc9e7166276fc not found: ID does not exist"
Sep 11 04:01:55.749398 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:55.749346    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" podStartSLOduration=1.749334929 podStartE2EDuration="1.749334929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:55.749232326 +0000 UTC m=+713.615252060" watchObservedRunningTime="2026-09-11 04:01:55.749334929 +0000 UTC m=+713.615354728"
Sep 11 04:01:55.753102 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:55.753073    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-600aef\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="1da2ffe3-253f-4a71-a33d-09ee88c16ad2" pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:56.342061 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:56.342023    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:56.347980 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:56.347955    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:56.355042 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:56.355006    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-600aef\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="1da2ffe3-253f-4a71-a33d-09ee88c16ad2" pod="opendatahub/test-curl-7498-600aef"
Sep 11 04:01:56.612234 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:56.612159    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da2ffe3-253f-4a71-a33d-09ee88c16ad2" path="/var/lib/kubelet/pods/1da2ffe3-253f-4a71-a33d-09ee88c16ad2/volumes"
Sep 11 04:01:56.714119 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:56.714094    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:56.715099 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:56.715082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:01:56.872200 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:56.872147    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:56.968585 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:56.968562    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf5lr\" (UniqueName: \"kubernetes.io/projected/56e3e3da-c795-424a-88bc-5ff2940cc816-kube-api-access-sf5lr\") pod \"56e3e3da-c795-424a-88bc-5ff2940cc816\" (UID: \"56e3e3da-c795-424a-88bc-5ff2940cc816\") "
Sep 11 04:01:56.970472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:56.970452    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e3e3da-c795-424a-88bc-5ff2940cc816-kube-api-access-sf5lr" (OuterVolumeSpecName: "kube-api-access-sf5lr") pod "56e3e3da-c795-424a-88bc-5ff2940cc816" (UID: "56e3e3da-c795-424a-88bc-5ff2940cc816"). InnerVolumeSpecName "kube-api-access-sf5lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:01:57.070136 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:57.070106    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf5lr\" (UniqueName: \"kubernetes.io/projected/56e3e3da-c795-424a-88bc-5ff2940cc816-kube-api-access-sf5lr\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:01:57.719141 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:57.719103    2566 generic.go:356] "Generic (PLEG): container finished" podID="56e3e3da-c795-424a-88bc-5ff2940cc816" containerID="b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a" exitCode=137
Sep 11 04:01:57.719141 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:57.719142    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:57.719675 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:57.719208    2566 scope.go:117] "RemoveContainer" containerID="b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a"
Sep 11 04:01:57.727615 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:57.727600    2566 scope.go:117] "RemoveContainer" containerID="b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a"
Sep 11 04:01:57.727891 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:57.727874    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a\": container with ID starting with b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a not found: ID does not exist" containerID="b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a"
Sep 11 04:01:57.727945 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:57.727901    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a"} err="failed to get container status \"b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a\": rpc error: code = NotFound desc = could not find container \"b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a\": container with ID starting with b570fc2a0bd10121d5fef008415d822a85b5b1a3bf1098f4a4eb6c55e904553a not found: ID does not exist"
Sep 11 04:01:57.729742 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:57.729722    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-e78f30\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="56e3e3da-c795-424a-88bc-5ff2940cc816" pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:58.108187 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.108106    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"]
Sep 11 04:01:58.113204 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.113184    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.115266 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:58.115240    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-e78f30\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="56e3e3da-c795-424a-88bc-5ff2940cc816" pod="opendatahub/test-curl-7498-e78f30"
Sep 11 04:01:58.116039 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.116015    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f547da13-serving-cert\""
Sep 11 04:01:58.122946 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.122923    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"]
Sep 11 04:01:58.281012 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.280981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9p8\" (UniqueName: \"kubernetes.io/projected/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-kube-api-access-2r9p8\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.281163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.281016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-metrics-certs\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.281163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.281093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-openshift-service-ca\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.281163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.281153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.382213 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.382139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-openshift-service-ca\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.382213 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.382186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.382418 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.382227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9p8\" (UniqueName: \"kubernetes.io/projected/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-kube-api-access-2r9p8\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.382418 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.382246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-metrics-certs\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.382418 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:58.382361    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-serving-cert: secret "maas-api-e2e-ait-f547da13-serving-cert" not found
Sep 11 04:01:58.382569 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:01:58.382453    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls podName:7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba nodeName:}" failed. No retries permitted until 2026-09-11 04:01:58.882431904 +0000 UTC m=+716.748451639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls") pod "maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9" (UID: "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba") : secret "maas-api-e2e-ait-f547da13-serving-cert" not found
Sep 11 04:01:58.382824 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.382801    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-openshift-service-ca\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.384643 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.384622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-metrics-certs\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.394385 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.394365    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9p8\" (UniqueName: \"kubernetes.io/projected/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-kube-api-access-2r9p8\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.612879 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.612847    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e3e3da-c795-424a-88bc-5ff2940cc816" path="/var/lib/kubelet/pods/56e3e3da-c795-424a-88bc-5ff2940cc816/volumes"
Sep 11 04:01:58.886292 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.886251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:58.889076 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:58.889049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls\") pod \"maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:59.026205 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.026173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:01:59.319935 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.319902    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"]
Sep 11 04:01:59.323655 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.323633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.325951 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.325932    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"]
Sep 11 04:01:59.326120 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.326099    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-944e93-serving-cert\""
Sep 11 04:01:59.329279 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:01:59.329257    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6a33dc_1ade_4e3a_86ea_14b4f9cd3cba.slice/crio-0b3f7eaefdff1a4fcbf4e0badb6b875dcebf9d86120baff7a77634e81008fd79 WatchSource:0}: Error finding container 0b3f7eaefdff1a4fcbf4e0badb6b875dcebf9d86120baff7a77634e81008fd79: Status 404 returned error can't find the container with id 0b3f7eaefdff1a4fcbf4e0badb6b875dcebf9d86120baff7a77634e81008fd79
Sep 11 04:01:59.347576 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.347537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"]
Sep 11 04:01:59.492135 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.492101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44x2\" (UniqueName: \"kubernetes.io/projected/71bfae8e-1c6e-47a8-add1-e25ba79726be-kube-api-access-d44x2\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.492305 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.492148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-maas-api-tls\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.492305 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.492168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bfae8e-1c6e-47a8-add1-e25ba79726be-openshift-service-ca\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.492383 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.492293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-metrics-certs\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.593749 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.593672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44x2\" (UniqueName: \"kubernetes.io/projected/71bfae8e-1c6e-47a8-add1-e25ba79726be-kube-api-access-d44x2\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.593749 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.593736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-maas-api-tls\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.593934 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.593762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bfae8e-1c6e-47a8-add1-e25ba79726be-openshift-service-ca\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.593934 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.593809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-metrics-certs\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.594382 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.594360    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bfae8e-1c6e-47a8-add1-e25ba79726be-openshift-service-ca\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.596132 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.596108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-metrics-certs\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.596215 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.596147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-maas-api-tls\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.606274 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.606253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44x2\" (UniqueName: \"kubernetes.io/projected/71bfae8e-1c6e-47a8-add1-e25ba79726be-kube-api-access-d44x2\") pod \"maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.636145 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.636123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:01:59.730861 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.730841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9" event={"ID":"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba","Type":"ContainerStarted","Data":"0b3f7eaefdff1a4fcbf4e0badb6b875dcebf9d86120baff7a77634e81008fd79"}
Sep 11 04:01:59.911169 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:01:59.911138    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"]
Sep 11 04:01:59.913275 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:01:59.913249    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bfae8e_1c6e_47a8_add1_e25ba79726be.slice/crio-47a55925d924c70c594005429f512d0a9c050eecabc7a3877f511600867e9e1e WatchSource:0}: Error finding container 47a55925d924c70c594005429f512d0a9c050eecabc7a3877f511600867e9e1e: Status 404 returned error can't find the container with id 47a55925d924c70c594005429f512d0a9c050eecabc7a3877f511600867e9e1e
Sep 11 04:02:00.189394 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.189317    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"]
Sep 11 04:02:00.192945 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.192928    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:00.195455 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.195434    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-944e93-dockercfg-52nxt\""
Sep 11 04:02:00.195548 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.195480    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-944e93\""
Sep 11 04:02:00.202077 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.202054    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"]
Sep 11 04:02:00.300017 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.299992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9\" (UID: \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:00.300156 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.300057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwc4n\" (UniqueName: \"kubernetes.io/projected/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-kube-api-access-gwc4n\") pod \"payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9\" (UID: \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:00.401621 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.401589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9\" (UID: \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:00.401760 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.401671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwc4n\" (UniqueName: \"kubernetes.io/projected/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-kube-api-access-gwc4n\") pod \"payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9\" (UID: \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:00.402249 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.402230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9\" (UID: \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:00.415467 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.415436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwc4n\" (UniqueName: \"kubernetes.io/projected/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-kube-api-access-gwc4n\") pod \"payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9\" (UID: \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:00.503982 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.503952    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:00.736942 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.736908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f" event={"ID":"71bfae8e-1c6e-47a8-add1-e25ba79726be","Type":"ContainerStarted","Data":"d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c"}
Sep 11 04:02:00.737123 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.736969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f" event={"ID":"71bfae8e-1c6e-47a8-add1-e25ba79726be","Type":"ContainerStarted","Data":"47a55925d924c70c594005429f512d0a9c050eecabc7a3877f511600867e9e1e"}
Sep 11 04:02:00.737123 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.737024    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:02:00.738573 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.738549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9" event={"ID":"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba","Type":"ContainerStarted","Data":"ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b"}
Sep 11 04:02:00.738670 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.738636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:02:00.761634 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.761555    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f" podStartSLOduration=1.455598574 podStartE2EDuration="1.761540128s" totalImagesPullingTime="305.941554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:00.759148492 +0000 UTC m=+718.625168226" watchObservedRunningTime="2026-09-11 04:02:00.761540128 +0000 UTC m=+718.627559911"
Sep 11 04:02:00.775172 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.775153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"]
Sep 11 04:02:00.777283 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:02:00.777259    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2687d39_dbd8_43ca_8eba_e1b7205c84f2.slice/crio-86d9c10d2bc4de283a2c3b88443a17361e99d6f874d9b4ba14c9989ac3f6ae57 WatchSource:0}: Error finding container 86d9c10d2bc4de283a2c3b88443a17361e99d6f874d9b4ba14c9989ac3f6ae57: Status 404 returned error can't find the container with id 86d9c10d2bc4de283a2c3b88443a17361e99d6f874d9b4ba14c9989ac3f6ae57
Sep 11 04:02:00.777842 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:00.777798    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9" podStartSLOduration=2.433886207 podStartE2EDuration="2.777779677s" totalImagesPullingTime="343.89347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:00.775948893 +0000 UTC m=+718.641968628" watchObservedRunningTime="2026-09-11 04:02:00.777779677 +0000 UTC m=+718.643799413"
Sep 11 04:02:01.311299 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:01.311263    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:02:01.745143 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:01.745108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9" event={"ID":"c2687d39-dbd8-43ca-8eba-e1b7205c84f2","Type":"ContainerStarted","Data":"b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644"}
Sep 11 04:02:01.745322 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:01.745170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9" event={"ID":"c2687d39-dbd8-43ca-8eba-e1b7205c84f2","Type":"ContainerStarted","Data":"86d9c10d2bc4de283a2c3b88443a17361e99d6f874d9b4ba14c9989ac3f6ae57"}
Sep 11 04:02:01.764347 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:01.764283    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9" podStartSLOduration=1.7642689539999998 podStartE2EDuration="1.764268954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:01.76333827 +0000 UTC m=+719.629358006" watchObservedRunningTime="2026-09-11 04:02:01.764268954 +0000 UTC m=+719.630288721"
Sep 11 04:02:02.755961 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:02.755929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:03.688507 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:03.688476    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:02:06.750768 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:06.750741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:02:06.751390 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:06.751370    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:02:13.764622 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:13.764586    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:02:14.603856 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.603820    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"]
Sep 11 04:02:14.612574 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.612548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.615386 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.615365    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5d685d-openshift-default-dockercfg-jkpzc\""
Sep 11 04:02:14.621751 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.621727    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"]
Sep 11 04:02:14.723308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-envoy\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.723543 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-podinfo\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.723543 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-credential-socket\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.723543 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-socket\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.723543 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-token\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.723718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz84x\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-kube-api-access-xz84x\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.723718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-certs\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.723801 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188faf8b-3a45-4bf6-8603-a83ac299acd1-istiod-ca-cert\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.723801 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.723771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-data\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.824836 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.824802    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-socket\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.824845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-token\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.824865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz84x\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-kube-api-access-xz84x\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.824937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-certs\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.824984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188faf8b-3a45-4bf6-8603-a83ac299acd1-istiod-ca-cert\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-data\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-envoy\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-podinfo\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825654 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-socket\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825654 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-credential-socket\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825654 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-certs\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825654 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-data\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825654 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-credential-socket\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.825885 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.825669    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188faf8b-3a45-4bf6-8603-a83ac299acd1-istiod-ca-cert\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.827368 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.827336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-envoy\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.827508 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.827437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-podinfo\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.832742 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.832714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-token\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.832968 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.832950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz84x\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-kube-api-access-xz84x\") pod \"e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") " pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:14.924614 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:14.924545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:15.227173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:15.227146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"]
Sep 11 04:02:15.229070 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:02:15.229045    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188faf8b_3a45_4bf6_8603_a83ac299acd1.slice/crio-4eb8e2e546f08dc7c178cb841ea1d2db0797e51530b81f51d695f380bcb97744 WatchSource:0}: Error finding container 4eb8e2e546f08dc7c178cb841ea1d2db0797e51530b81f51d695f380bcb97744: Status 404 returned error can't find the container with id 4eb8e2e546f08dc7c178cb841ea1d2db0797e51530b81f51d695f380bcb97744
Sep 11 04:02:15.231451 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:15.231423    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:15.231558 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:15.231505    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:15.231558 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:15.231552    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:15.828969 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:15.828925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" event={"ID":"188faf8b-3a45-4bf6-8603-a83ac299acd1","Type":"ContainerStarted","Data":"598ece731d00575b9ba2f482a7b8d641c6c75b1a8c5e6de180996219c2a5b5fa"}
Sep 11 04:02:15.829441 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:15.828994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" event={"ID":"188faf8b-3a45-4bf6-8603-a83ac299acd1","Type":"ContainerStarted","Data":"4eb8e2e546f08dc7c178cb841ea1d2db0797e51530b81f51d695f380bcb97744"}
Sep 11 04:02:15.852460 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:15.852379    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" podStartSLOduration=1.852357711 podStartE2EDuration="1.852357711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:15.850071827 +0000 UTC m=+733.716091563" watchObservedRunningTime="2026-09-11 04:02:15.852357711 +0000 UTC m=+733.718377447"
Sep 11 04:02:15.925290 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:15.925261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:16.551372 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:16.551340    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:02:16.930511 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:16.930421    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:17.840551 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:17.840518    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:17.841714 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:17.841689    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:19.856049 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:19.856019    2566 generic.go:356] "Generic (PLEG): container finished" podID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerID="32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329" exitCode=0
Sep 11 04:02:19.856504 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:19.856094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" event={"ID":"b6208c03-d739-42c9-a0e0-2521d4907b94","Type":"ContainerDied","Data":"32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329"}
Sep 11 04:02:19.856621 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:19.856603    2566 scope.go:117] "RemoveContainer" containerID="32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329"
Sep 11 04:02:19.857820 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:19.857786    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:19.857905 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:19.857875    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:19.857967 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:19.857919    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:20.865852 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:20.865815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" event={"ID":"b6208c03-d739-42c9-a0e0-2521d4907b94","Type":"ContainerStarted","Data":"a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d"}
Sep 11 04:02:21.342294 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:21.342256    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:02:21.342739 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:21.342709    2566 patch_prober.go:28] interesting pod/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 04:02:21.342845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:21.342777    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 04:02:22.342254 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:22.342208    2566 patch_prober.go:28] interesting pod/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 04:02:22.342754 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:22.342283    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 04:02:23.342605 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:23.342569    2566 patch_prober.go:28] interesting pod/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 04:02:23.343180 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:23.343147    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 04:02:24.341965 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:24.341924    2566 patch_prober.go:28] interesting pod/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 04:02:24.342162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:24.341996    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 04:02:25.341963 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:25.341927    2566 patch_prober.go:28] interesting pod/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 04:02:25.342484 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:25.341997    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 04:02:26.345750 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:26.345721    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:02:26.346230 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:26.346204    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:02:26.346921 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:26.346902    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:02:33.283356 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:33.283320    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:02:35.945470 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:35.945358    2566 generic.go:356] "Generic (PLEG): container finished" podID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerID="598ece731d00575b9ba2f482a7b8d641c6c75b1a8c5e6de180996219c2a5b5fa" exitCode=0
Sep 11 04:02:35.945470 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:35.945432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" event={"ID":"188faf8b-3a45-4bf6-8603-a83ac299acd1","Type":"ContainerDied","Data":"598ece731d00575b9ba2f482a7b8d641c6c75b1a8c5e6de180996219c2a5b5fa"}
Sep 11 04:02:35.945910 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:35.945797    2566 scope.go:117] "RemoveContainer" containerID="598ece731d00575b9ba2f482a7b8d641c6c75b1a8c5e6de180996219c2a5b5fa"
Sep 11 04:02:35.946788 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:35.946763    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:35.946839 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:35.946829    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:35.946876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:35.946860    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:36.956480 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:36.956437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" event={"ID":"188faf8b-3a45-4bf6-8603-a83ac299acd1","Type":"ContainerStarted","Data":"d24f6666a4577594df0e32b935c47a95695f791af2ffb0a69565eaec9cef0f11"}
Sep 11 04:02:37.925597 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:37.925563    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:37.925773 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:37.925716    2566 patch_prober.go:28] interesting pod/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 11 04:02:37.925818 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:37.925767    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 11 04:02:38.925626 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:38.925594    2566 patch_prober.go:28] interesting pod/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 11 04:02:38.925996 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:38.925670    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 11 04:02:38.954430 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:38.953488    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"]
Sep 11 04:02:38.954430 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:38.954054    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9" podUID="7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" containerName="maas-api" containerID="cri-o://ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b" gracePeriod=30
Sep 11 04:02:39.272577 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.272543    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:02:39.372720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.372680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-metrics-certs\") pod \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") "
Sep 11 04:02:39.373078 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.373056    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-openshift-service-ca\") pod \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") "
Sep 11 04:02:39.373229 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.373213    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r9p8\" (UniqueName: \"kubernetes.io/projected/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-kube-api-access-2r9p8\") pod \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") "
Sep 11 04:02:39.373341 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.373328    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls\") pod \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\" (UID: \"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba\") "
Sep 11 04:02:39.374571 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.374264    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" (UID: "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:02:39.374571 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.374383    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-openshift-service-ca\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:02:39.379049 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.379021    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-kube-api-access-2r9p8" (OuterVolumeSpecName: "kube-api-access-2r9p8") pod "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" (UID: "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba"). InnerVolumeSpecName "kube-api-access-2r9p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:02:39.380059 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.380036    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" (UID: "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:02:39.380687 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.380669    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" (UID: "7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:02:39.474993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.474956    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r9p8\" (UniqueName: \"kubernetes.io/projected/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-kube-api-access-2r9p8\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:02:39.474993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.474987    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-maas-api-tls\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:02:39.474993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.474999    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba-metrics-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:02:39.925196 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.925156    2566 patch_prober.go:28] interesting pod/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 11 04:02:39.925357 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.925230    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 11 04:02:39.982839 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.982800    2566 generic.go:356] "Generic (PLEG): container finished" podID="7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" containerID="ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b" exitCode=0
Sep 11 04:02:39.983332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.982847    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"
Sep 11 04:02:39.983332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.982850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9" event={"ID":"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba","Type":"ContainerDied","Data":"ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b"}
Sep 11 04:02:39.983332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.982916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9" event={"ID":"7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba","Type":"ContainerDied","Data":"0b3f7eaefdff1a4fcbf4e0badb6b875dcebf9d86120baff7a77634e81008fd79"}
Sep 11 04:02:39.983332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.982949    2566 scope.go:117] "RemoveContainer" containerID="ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b"
Sep 11 04:02:39.996214 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.996195    2566 scope.go:117] "RemoveContainer" containerID="ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b"
Sep 11 04:02:39.996548 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:02:39.996528    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b\": container with ID starting with ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b not found: ID does not exist" containerID="ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b"
Sep 11 04:02:39.996621 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:39.996558    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b"} err="failed to get container status \"ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b\": rpc error: code = NotFound desc = could not find container \"ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b\": container with ID starting with ca608c9976771d681dcf8467a844e95fd07bfaa51de71b4f5045faf52a80c86b not found: ID does not exist"
Sep 11 04:02:40.006893 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.006865    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"]
Sep 11 04:02:40.015162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.015135    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f547da13-9d9c6b7b5-v5rl9"]
Sep 11 04:02:40.283668 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.283628    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"]
Sep 11 04:02:40.284130 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.284114    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" containerName="maas-api"
Sep 11 04:02:40.284209 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.284133    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" containerName="maas-api"
Sep 11 04:02:40.284264 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.284224    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" containerName="maas-api"
Sep 11 04:02:40.296497 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.296464    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.299617 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.299596    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d3676176-gw-openshift-default-dockercfg-stm45\""
Sep 11 04:02:40.305826 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.305806    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"]
Sep 11 04:02:40.389988 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.389954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xv6n\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-kube-api-access-6xv6n\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.389988 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.389994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-istio-token\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.390216 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.390017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-socket\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.390216 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.390056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-envoy\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.390216 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.390076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8965f687-69b6-42a5-bf30-c549c4064a95-istiod-ca-cert\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.390216 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.390104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-data\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.390216 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.390126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8965f687-69b6-42a5-bf30-c549c4064a95-istio-podinfo\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.390400 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.390229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-credential-socket\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.390400 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.390259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-certs\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.491568 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.491531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-credential-socket\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.491769 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.491581    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-certs\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.491769 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.491666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xv6n\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-kube-api-access-6xv6n\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.491769 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.491698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-istio-token\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.491769 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.491725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-socket\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.492005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-credential-socket\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492085 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.491998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-envoy\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492085 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.492070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8965f687-69b6-42a5-bf30-c549c4064a95-istiod-ca-cert\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.492017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-certs\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.492097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-data\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.492104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-socket\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.492136    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8965f687-69b6-42a5-bf30-c549c4064a95-istio-podinfo\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492361 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.492287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-data\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.492669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.492643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8965f687-69b6-42a5-bf30-c549c4064a95-istiod-ca-cert\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.494204 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.494175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-envoy\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.494489 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.494471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8965f687-69b6-42a5-bf30-c549c4064a95-istio-podinfo\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.506587 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.506563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xv6n\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-kube-api-access-6xv6n\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.506709 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.506682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-istio-token\") pod \"e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") " pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.608812 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.608724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:40.612951 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.612728    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba" path="/var/lib/kubelet/pods/7d6a33dc-1ade-4e3a-86ea-14b4f9cd3cba/volumes"
Sep 11 04:02:40.913187 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.913154    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"]
Sep 11 04:02:40.915134 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:02:40.915103    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8965f687_69b6_42a5_bf30_c549c4064a95.slice/crio-918461bfed82a1d2790315ca44ace5d47d8b6a889594b0f78528c98330cbb5a0 WatchSource:0}: Error finding container 918461bfed82a1d2790315ca44ace5d47d8b6a889594b0f78528c98330cbb5a0: Status 404 returned error can't find the container with id 918461bfed82a1d2790315ca44ace5d47d8b6a889594b0f78528c98330cbb5a0
Sep 11 04:02:40.917362 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.917328    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:40.917472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.917393    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:40.917472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.917435    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:40.925183 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.925159    2566 patch_prober.go:28] interesting pod/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 11 04:02:40.925284 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.925208    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 11 04:02:40.990639 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:40.990613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" event={"ID":"8965f687-69b6-42a5-bf30-c549c4064a95","Type":"ContainerStarted","Data":"918461bfed82a1d2790315ca44ace5d47d8b6a889594b0f78528c98330cbb5a0"}
Sep 11 04:02:41.924837 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:41.924803    2566 patch_prober.go:28] interesting pod/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 11 04:02:41.924978 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:41.924855    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 11 04:02:41.997521 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:41.997489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" event={"ID":"8965f687-69b6-42a5-bf30-c549c4064a95","Type":"ContainerStarted","Data":"6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4"}
Sep 11 04:02:42.017388 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:42.017322    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podStartSLOduration=2.017302917 podStartE2EDuration="2.017302917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:42.016851253 +0000 UTC m=+759.882871000" watchObservedRunningTime="2026-09-11 04:02:42.017302917 +0000 UTC m=+759.883322655"
Sep 11 04:02:42.613079 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:42.613039    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:42.613774 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:42.613759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:42.928517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:42.928433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:42.928816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:42.928797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:42.929421 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:42.929392    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:02:43.002439 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:43.002392    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:43.003348 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:43.003327    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:44.345905 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.345870    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"]
Sep 11 04:02:44.349505 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.349485    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:44.357298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.357277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d3676176-dockercfg-v5gqf\""
Sep 11 04:02:44.357420 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.357341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d3676176\""
Sep 11 04:02:44.364437 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.364382    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"]
Sep 11 04:02:44.530330 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.530297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cebf6af-c228-4782-9e69-47cd18c489df-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7\" (UID: \"0cebf6af-c228-4782-9e69-47cd18c489df\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:44.530330 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.530330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfvnm\" (UniqueName: \"kubernetes.io/projected/0cebf6af-c228-4782-9e69-47cd18c489df-kube-api-access-dfvnm\") pod \"payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7\" (UID: \"0cebf6af-c228-4782-9e69-47cd18c489df\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:44.631789 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.631693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cebf6af-c228-4782-9e69-47cd18c489df-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7\" (UID: \"0cebf6af-c228-4782-9e69-47cd18c489df\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:44.631789 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.631737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfvnm\" (UniqueName: \"kubernetes.io/projected/0cebf6af-c228-4782-9e69-47cd18c489df-kube-api-access-dfvnm\") pod \"payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7\" (UID: \"0cebf6af-c228-4782-9e69-47cd18c489df\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:44.632575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.632551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cebf6af-c228-4782-9e69-47cd18c489df-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7\" (UID: \"0cebf6af-c228-4782-9e69-47cd18c489df\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:44.639568 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.639538    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfvnm\" (UniqueName: \"kubernetes.io/projected/0cebf6af-c228-4782-9e69-47cd18c489df-kube-api-access-dfvnm\") pod \"payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7\" (UID: \"0cebf6af-c228-4782-9e69-47cd18c489df\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:44.660744 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.660718    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:44.930484 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:44.930462    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"]
Sep 11 04:02:44.932538 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:02:44.932510    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cebf6af_c228_4782_9e69_47cd18c489df.slice/crio-0f63153c61f0b8c04f3a833d28757194405dfed14279253bc71663682890a715 WatchSource:0}: Error finding container 0f63153c61f0b8c04f3a833d28757194405dfed14279253bc71663682890a715: Status 404 returned error can't find the container with id 0f63153c61f0b8c04f3a833d28757194405dfed14279253bc71663682890a715
Sep 11 04:02:45.014476 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:45.014447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7" event={"ID":"0cebf6af-c228-4782-9e69-47cd18c489df","Type":"ContainerStarted","Data":"8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a"}
Sep 11 04:02:45.014606 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:45.014505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7" event={"ID":"0cebf6af-c228-4782-9e69-47cd18c489df","Type":"ContainerStarted","Data":"0f63153c61f0b8c04f3a833d28757194405dfed14279253bc71663682890a715"}
Sep 11 04:02:45.014778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:45.014765    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:45.030055 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:45.030005    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7" podStartSLOduration=1.029990033 podStartE2EDuration="1.029990033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:45.029196963 +0000 UTC m=+762.895216710" watchObservedRunningTime="2026-09-11 04:02:45.029990033 +0000 UTC m=+762.896009769"
Sep 11 04:02:47.183112 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:47.183077    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:02:56.024924 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:56.024891    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:02:58.003484 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:58.003449    2566 patch_prober.go:28] interesting pod/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 04:02:58.003953 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:58.003506    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 04:02:58.100822 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:58.100774    2566 generic.go:356] "Generic (PLEG): container finished" podID="8965f687-69b6-42a5-bf30-c549c4064a95" containerID="6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4" exitCode=0
Sep 11 04:02:58.100995 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:58.100846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" event={"ID":"8965f687-69b6-42a5-bf30-c549c4064a95","Type":"ContainerDied","Data":"6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4"}
Sep 11 04:02:58.101260 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:58.101247    2566 scope.go:117] "RemoveContainer" containerID="6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4"
Sep 11 04:02:58.102337 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:58.102302    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:58.102458 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:58.102396    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:58.102532 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:58.102462    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:02:59.107456 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:59.107395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" event={"ID":"8965f687-69b6-42a5-bf30-c549c4064a95","Type":"ContainerStarted","Data":"eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65"}
Sep 11 04:02:59.609126 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:59.609085    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:02:59.609323 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:59.609300    2566 patch_prober.go:28] interesting pod/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 04:02:59.609370 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:02:59.609353    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 04:03:00.609123 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:00.609088    2566 patch_prober.go:28] interesting pod/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 04:03:00.609547 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:00.609145    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 04:03:01.086635 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:01.086600    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:03:01.609718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:01.609685    2566 patch_prober.go:28] interesting pod/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 04:03:01.610095 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:01.609739    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 04:03:02.609343 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:02.609305    2566 patch_prober.go:28] interesting pod/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 04:03:02.610049 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:02.609401    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 04:03:03.609851 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:03.609822    2566 patch_prober.go:28] interesting pod/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 04:03:03.610016 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:03.609886    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 04:03:04.631578 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:04.631551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:03:04.632207 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:04.632042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:03:04.633334 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:04.633313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:03:14.252938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.252899    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"]
Sep 11 04:03:14.253813 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.253770    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7" podUID="0cebf6af-c228-4782-9e69-47cd18c489df" containerName="payload-processing" containerID="cri-o://8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a" gracePeriod=30
Sep 11 04:03:14.515083 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.513866    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:03:14.541394 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.541324    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"]
Sep 11 04:03:14.541923 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.541594    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy" containerID="cri-o://eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65" gracePeriod=30
Sep 11 04:03:14.609177 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.609149    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfvnm\" (UniqueName: \"kubernetes.io/projected/0cebf6af-c228-4782-9e69-47cd18c489df-kube-api-access-dfvnm\") pod \"0cebf6af-c228-4782-9e69-47cd18c489df\" (UID: \"0cebf6af-c228-4782-9e69-47cd18c489df\") "
Sep 11 04:03:14.609366 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.609238    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cebf6af-c228-4782-9e69-47cd18c489df-plugins-config-volume\") pod \"0cebf6af-c228-4782-9e69-47cd18c489df\" (UID: \"0cebf6af-c228-4782-9e69-47cd18c489df\") "
Sep 11 04:03:14.609622 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.609596    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cebf6af-c228-4782-9e69-47cd18c489df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0cebf6af-c228-4782-9e69-47cd18c489df" (UID: "0cebf6af-c228-4782-9e69-47cd18c489df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:14.611257 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.611234    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cebf6af-c228-4782-9e69-47cd18c489df-kube-api-access-dfvnm" (OuterVolumeSpecName: "kube-api-access-dfvnm") pod "0cebf6af-c228-4782-9e69-47cd18c489df" (UID: "0cebf6af-c228-4782-9e69-47cd18c489df"). InnerVolumeSpecName "kube-api-access-dfvnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:14.710290 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.710248    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfvnm\" (UniqueName: \"kubernetes.io/projected/0cebf6af-c228-4782-9e69-47cd18c489df-kube-api-access-dfvnm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:14.710290 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:14.710284    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cebf6af-c228-4782-9e69-47cd18c489df-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:15.196295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.196260    2566 generic.go:356] "Generic (PLEG): container finished" podID="0cebf6af-c228-4782-9e69-47cd18c489df" containerID="8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a" exitCode=0
Sep 11 04:03:15.196540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.196307    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"
Sep 11 04:03:15.196540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.196349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7" event={"ID":"0cebf6af-c228-4782-9e69-47cd18c489df","Type":"ContainerDied","Data":"8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a"}
Sep 11 04:03:15.196540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.196436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7" event={"ID":"0cebf6af-c228-4782-9e69-47cd18c489df","Type":"ContainerDied","Data":"0f63153c61f0b8c04f3a833d28757194405dfed14279253bc71663682890a715"}
Sep 11 04:03:15.196540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.196475    2566 scope.go:117] "RemoveContainer" containerID="8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a"
Sep 11 04:03:15.208343 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.208315    2566 scope.go:117] "RemoveContainer" containerID="8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a"
Sep 11 04:03:15.208712 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:03:15.208686    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a\": container with ID starting with 8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a not found: ID does not exist" containerID="8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a"
Sep 11 04:03:15.208873 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.208723    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a"} err="failed to get container status \"8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a\": rpc error: code = NotFound desc = could not find container \"8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a\": container with ID starting with 8f90cc46085453b5664b429837493e5a042d4522c320191c4ed5b022a437165a not found: ID does not exist"
Sep 11 04:03:15.225516 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.225486    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"]
Sep 11 04:03:15.231185 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:15.231158    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3676176-7c8dbd956b-r4rm7"]
Sep 11 04:03:16.482715 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:16.482673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:03:16.612994 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:16.612964    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cebf6af-c228-4782-9e69-47cd18c489df" path="/var/lib/kubelet/pods/0cebf6af-c228-4782-9e69-47cd18c489df/volumes"
Sep 11 04:03:19.633578 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.633542    2566 patch_prober.go:28] interesting pod/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 04:03:19.633928 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.633610    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 04:03:19.811797 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.811737    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:03:19.958291 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958265    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-data\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958481 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958306    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-envoy\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958481 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8965f687-69b6-42a5-bf30-c549c4064a95-istio-podinfo\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958481 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958381    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-certs\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958481 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958466    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8965f687-69b6-42a5-bf30-c549c4064a95-istiod-ca-cert\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958491    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-credential-socket\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958517    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-data" (OuterVolumeSpecName: "istio-data") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:19.958721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xv6n\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-kube-api-access-6xv6n\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-istio-token\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958606    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-socket\") pod \"8965f687-69b6-42a5-bf30-c549c4064a95\" (UID: \"8965f687-69b6-42a5-bf30-c549c4064a95\") "
Sep 11 04:03:19.958998 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958718    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:19.958998 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958736    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:19.958998 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8965f687-69b6-42a5-bf30-c549c4064a95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:19.958998 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.958904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:19.959214 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.959006    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8965f687-69b6-42a5-bf30-c549c4064a95-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:19.959214 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.959022    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:19.959214 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.959036    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:19.959214 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.959049    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:19.959214 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.959062    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:19.960802 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.960770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-kube-api-access-6xv6n" (OuterVolumeSpecName: "kube-api-access-6xv6n") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "kube-api-access-6xv6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:19.961060 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.961040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:03:19.961060 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.961040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8965f687-69b6-42a5-bf30-c549c4064a95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:03:19.961232 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:19.961048    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-istio-token" (OuterVolumeSpecName: "istio-token") pod "8965f687-69b6-42a5-bf30-c549c4064a95" (UID: "8965f687-69b6-42a5-bf30-c549c4064a95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:20.059916 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.059881    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xv6n\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-kube-api-access-6xv6n\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:20.059916 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.059911    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8965f687-69b6-42a5-bf30-c549c4064a95-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:20.060100 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.059926    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8965f687-69b6-42a5-bf30-c549c4064a95-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:20.060100 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.059935    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8965f687-69b6-42a5-bf30-c549c4064a95-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:20.066628 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.066565    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"]
Sep 11 04:03:20.066990 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.066978    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cebf6af-c228-4782-9e69-47cd18c489df" containerName="payload-processing"
Sep 11 04:03:20.067031 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.066992    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cebf6af-c228-4782-9e69-47cd18c489df" containerName="payload-processing"
Sep 11 04:03:20.067031 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.067006    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy"
Sep 11 04:03:20.067031 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.067012    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy"
Sep 11 04:03:20.067031 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.067029    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy"
Sep 11 04:03:20.067157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.067035    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy"
Sep 11 04:03:20.067157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.067093    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cebf6af-c228-4782-9e69-47cd18c489df" containerName="payload-processing"
Sep 11 04:03:20.067157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.067101    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy"
Sep 11 04:03:20.067157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.067109    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" containerName="istio-proxy"
Sep 11 04:03:20.070449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.070434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:20.072855 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.072799    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b0d05f3\""
Sep 11 04:03:20.079305 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.079285    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"]
Sep 11 04:03:20.161367 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.161333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg\" (UID: \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:20.161520 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.161385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tvx\" (UniqueName: \"kubernetes.io/projected/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-kube-api-access-n2tvx\") pod \"payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg\" (UID: \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:20.223942 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.223908    2566 generic.go:356] "Generic (PLEG): container finished" podID="8965f687-69b6-42a5-bf30-c549c4064a95" containerID="eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65" exitCode=0
Sep 11 04:03:20.224162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.224000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" event={"ID":"8965f687-69b6-42a5-bf30-c549c4064a95","Type":"ContainerDied","Data":"eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65"}
Sep 11 04:03:20.224162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.224012    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"
Sep 11 04:03:20.224162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.224071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl" event={"ID":"8965f687-69b6-42a5-bf30-c549c4064a95","Type":"ContainerDied","Data":"918461bfed82a1d2790315ca44ace5d47d8b6a889594b0f78528c98330cbb5a0"}
Sep 11 04:03:20.224162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.224110    2566 scope.go:117] "RemoveContainer" containerID="eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65"
Sep 11 04:03:20.233175 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.233159    2566 scope.go:117] "RemoveContainer" containerID="6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4"
Sep 11 04:03:20.241130 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.241113    2566 scope.go:117] "RemoveContainer" containerID="eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65"
Sep 11 04:03:20.241398 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:03:20.241379    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65\": container with ID starting with eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65 not found: ID does not exist" containerID="eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65"
Sep 11 04:03:20.241498 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.241421    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65"} err="failed to get container status \"eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65\": rpc error: code = NotFound desc = could not find container \"eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65\": container with ID starting with eed25aef6a1ab2d6d13f447656915d407776b299ed2eb53b5728becf8eb07c65 not found: ID does not exist"
Sep 11 04:03:20.241498 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.241444    2566 scope.go:117] "RemoveContainer" containerID="6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4"
Sep 11 04:03:20.241757 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:03:20.241740    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4\": container with ID starting with 6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4 not found: ID does not exist" containerID="6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4"
Sep 11 04:03:20.241822 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.241760    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4"} err="failed to get container status \"6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4\": rpc error: code = NotFound desc = could not find container \"6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4\": container with ID starting with 6a5ec7c2cb2345a5df04bc7b8acbf6867db6f1c549d96ba960c5d751eac710b4 not found: ID does not exist"
Sep 11 04:03:20.247268 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.247247    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"]
Sep 11 04:03:20.254158 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.254138    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d3676176-gw-openshift-default-766ccf88fb-bwrzl"]
Sep 11 04:03:20.262596 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.262576    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg\" (UID: \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:20.262673 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.262615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tvx\" (UniqueName: \"kubernetes.io/projected/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-kube-api-access-n2tvx\") pod \"payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg\" (UID: \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:20.263272 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.263255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg\" (UID: \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:20.270486 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.270464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tvx\" (UniqueName: \"kubernetes.io/projected/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-kube-api-access-n2tvx\") pod \"payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg\" (UID: \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:20.381864 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.381787    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:20.612400 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.612367    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8965f687-69b6-42a5-bf30-c549c4064a95" path="/var/lib/kubelet/pods/8965f687-69b6-42a5-bf30-c549c4064a95/volumes"
Sep 11 04:03:20.665211 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:20.665185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"]
Sep 11 04:03:20.665962 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:03:20.665930    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc218d0d2_4b29_473e_b242_e2fc9c29fdb5.slice/crio-8d95f33707e09752d99e094e131ea2b01f5ef119ba4b29e35b709c5d55e706c5 WatchSource:0}: Error finding container 8d95f33707e09752d99e094e131ea2b01f5ef119ba4b29e35b709c5d55e706c5: Status 404 returned error can't find the container with id 8d95f33707e09752d99e094e131ea2b01f5ef119ba4b29e35b709c5d55e706c5
Sep 11 04:03:21.231306 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:21.231269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg" event={"ID":"c218d0d2-4b29-473e-b242-e2fc9c29fdb5","Type":"ContainerStarted","Data":"a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8"}
Sep 11 04:03:21.231500 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:21.231327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg" event={"ID":"c218d0d2-4b29-473e-b242-e2fc9c29fdb5","Type":"ContainerStarted","Data":"8d95f33707e09752d99e094e131ea2b01f5ef119ba4b29e35b709c5d55e706c5"}
Sep 11 04:03:21.231575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:21.231511    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:21.247610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:21.247561    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg" podStartSLOduration=1.247547332 podStartE2EDuration="1.247547332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:21.245745344 +0000 UTC m=+799.111765080" watchObservedRunningTime="2026-09-11 04:03:21.247547332 +0000 UTC m=+799.113567068"
Sep 11 04:03:29.774023 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:29.773979    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:03:32.238575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:32.238544    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:42.378120 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.378087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:03:42.709229 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.709129    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"]
Sep 11 04:03:42.713037 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.713011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.717241 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.717208    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-944e93\"/\"default-dockercfg-lv9gs\""
Sep 11 04:03:42.717435 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.717294    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-944e93\"/\"kube-root-ca.crt\""
Sep 11 04:03:42.717435 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.717329    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-944e93\"/\"autores-model-944e93-kserve-self-signed-certs\""
Sep 11 04:03:42.717435 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.717358    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-944e93\"/\"openshift-service-ca.crt\""
Sep 11 04:03:42.724017 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.723987    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"]
Sep 11 04:03:42.766705 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.766632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tls-certs\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.767014 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.766770    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-model-cache\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.767014 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.766837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-home\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.767014 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.766886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tmp-dir\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.767014 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.766909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-dshm\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.767014 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.766951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47npv\" (UniqueName: \"kubernetes.io/projected/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-kube-api-access-47npv\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.868376 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.868306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tmp-dir\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.868376 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.868376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-dshm\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.868854 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.868423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47npv\" (UniqueName: \"kubernetes.io/projected/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-kube-api-access-47npv\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.868854 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.868553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tls-certs\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.868854 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.868702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-model-cache\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.868854 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.868848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-home\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.869069 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.868976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tmp-dir\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.869115 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.869086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-model-cache\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.869254 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.869226    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-home\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.871519 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.871493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-dshm\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.871757 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.871723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tls-certs\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:42.878454 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:42.878417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47npv\" (UniqueName: \"kubernetes.io/projected/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-kube-api-access-47npv\") pod \"autores-model-944e93-kserve-d848fbff-jjwtr\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") " pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:43.024064 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:43.024023    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:43.314108 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:43.314078    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"]
Sep 11 04:03:43.316244 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:03:43.316208    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a6ef017_27d2_4ea1_a5cb_ab84170c686e.slice/crio-c0740ea05b9dfba4580c0c41e19811ba7a4707be08bfe4044f0c20e92d7ea0cd WatchSource:0}: Error finding container c0740ea05b9dfba4580c0c41e19811ba7a4707be08bfe4044f0c20e92d7ea0cd: Status 404 returned error can't find the container with id c0740ea05b9dfba4580c0c41e19811ba7a4707be08bfe4044f0c20e92d7ea0cd
Sep 11 04:03:43.354919 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:43.354876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr" event={"ID":"7a6ef017-27d2-4ea1-a5cb-ab84170c686e","Type":"ContainerStarted","Data":"c0740ea05b9dfba4580c0c41e19811ba7a4707be08bfe4044f0c20e92d7ea0cd"}
Sep 11 04:03:44.363095 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:44.363056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr" event={"ID":"7a6ef017-27d2-4ea1-a5cb-ab84170c686e","Type":"ContainerStarted","Data":"568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae"}
Sep 11 04:03:44.378369 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:44.378316    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr" podStartSLOduration=2.378300595 podStartE2EDuration="2.378300595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:44.377681401 +0000 UTC m=+822.243701130" watchObservedRunningTime="2026-09-11 04:03:44.378300595 +0000 UTC m=+822.244320333"
Sep 11 04:03:49.874859 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:49.874824    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"]
Sep 11 04:03:49.875340 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:49.875072    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg" podUID="c218d0d2-4b29-473e-b242-e2fc9c29fdb5" containerName="payload-pre-processing" containerID="cri-o://a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8" gracePeriod=30
Sep 11 04:03:50.129162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.129108    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:50.237953 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.237927    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-plugins-config-volume\") pod \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\" (UID: \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\") "
Sep 11 04:03:50.238131 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.238038    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2tvx\" (UniqueName: \"kubernetes.io/projected/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-kube-api-access-n2tvx\") pod \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\" (UID: \"c218d0d2-4b29-473e-b242-e2fc9c29fdb5\") "
Sep 11 04:03:50.238346 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.238318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c218d0d2-4b29-473e-b242-e2fc9c29fdb5" (UID: "c218d0d2-4b29-473e-b242-e2fc9c29fdb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:03:50.239992 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.239967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-kube-api-access-n2tvx" (OuterVolumeSpecName: "kube-api-access-n2tvx") pod "c218d0d2-4b29-473e-b242-e2fc9c29fdb5" (UID: "c218d0d2-4b29-473e-b242-e2fc9c29fdb5"). InnerVolumeSpecName "kube-api-access-n2tvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:03:50.339073 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.339045    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:50.339073 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.339069    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2tvx\" (UniqueName: \"kubernetes.io/projected/c218d0d2-4b29-473e-b242-e2fc9c29fdb5-kube-api-access-n2tvx\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:03:50.394493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.394389    2566 generic.go:356] "Generic (PLEG): container finished" podID="c218d0d2-4b29-473e-b242-e2fc9c29fdb5" containerID="a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8" exitCode=0
Sep 11 04:03:50.394493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.394450    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"
Sep 11 04:03:50.394493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.394478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg" event={"ID":"c218d0d2-4b29-473e-b242-e2fc9c29fdb5","Type":"ContainerDied","Data":"a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8"}
Sep 11 04:03:50.394731 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.394539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg" event={"ID":"c218d0d2-4b29-473e-b242-e2fc9c29fdb5","Type":"ContainerDied","Data":"8d95f33707e09752d99e094e131ea2b01f5ef119ba4b29e35b709c5d55e706c5"}
Sep 11 04:03:50.394731 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.394571    2566 scope.go:117] "RemoveContainer" containerID="a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8"
Sep 11 04:03:50.406178 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.406162    2566 scope.go:117] "RemoveContainer" containerID="a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8"
Sep 11 04:03:50.406446 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:03:50.406427    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8\": container with ID starting with a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8 not found: ID does not exist" containerID="a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8"
Sep 11 04:03:50.406505 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.406453    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8"} err="failed to get container status \"a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8\": rpc error: code = NotFound desc = could not find container \"a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8\": container with ID starting with a037f63ce0f0829f6c6f23dcef509343f18ed71978343113f7e137b9270ae1c8 not found: ID does not exist"
Sep 11 04:03:50.416811 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.416780    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"]
Sep 11 04:03:50.422615 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.422594    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b0d05f3-564cfbcd86-lvlfg"]
Sep 11 04:03:50.613048 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:50.613019    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c218d0d2-4b29-473e-b242-e2fc9c29fdb5" path="/var/lib/kubelet/pods/c218d0d2-4b29-473e-b242-e2fc9c29fdb5/volumes"
Sep 11 04:03:53.024463 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:53.024433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:53.024463 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:53.024471    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:53.062480 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:53.062450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:53.422455 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:53.422356    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:03:56.097256 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.097227    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"]
Sep 11 04:03:56.097653 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.097615    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c218d0d2-4b29-473e-b242-e2fc9c29fdb5" containerName="payload-pre-processing"
Sep 11 04:03:56.097653 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.097627    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c218d0d2-4b29-473e-b242-e2fc9c29fdb5" containerName="payload-pre-processing"
Sep 11 04:03:56.097728 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.097697    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c218d0d2-4b29-473e-b242-e2fc9c29fdb5" containerName="payload-pre-processing"
Sep 11 04:03:56.102271 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.102252    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:56.104760 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.104737    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0e17d64c\""
Sep 11 04:03:56.105121 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.105101    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0e17d64c-dockercfg-t4999\""
Sep 11 04:03:56.112806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.112774    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"]
Sep 11 04:03:56.196711 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.196668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2df\" (UniqueName: \"kubernetes.io/projected/11e67fc5-e319-4793-9f85-b7d5d3909563-kube-api-access-8r2df\") pod \"payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh\" (UID: \"11e67fc5-e319-4793-9f85-b7d5d3909563\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:56.196884 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.196745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e67fc5-e319-4793-9f85-b7d5d3909563-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh\" (UID: \"11e67fc5-e319-4793-9f85-b7d5d3909563\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:56.297933 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.297894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e67fc5-e319-4793-9f85-b7d5d3909563-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh\" (UID: \"11e67fc5-e319-4793-9f85-b7d5d3909563\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:56.298097 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.297998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2df\" (UniqueName: \"kubernetes.io/projected/11e67fc5-e319-4793-9f85-b7d5d3909563-kube-api-access-8r2df\") pod \"payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh\" (UID: \"11e67fc5-e319-4793-9f85-b7d5d3909563\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:56.298608 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.298591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e67fc5-e319-4793-9f85-b7d5d3909563-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh\" (UID: \"11e67fc5-e319-4793-9f85-b7d5d3909563\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:56.308353 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.308320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2df\" (UniqueName: \"kubernetes.io/projected/11e67fc5-e319-4793-9f85-b7d5d3909563-kube-api-access-8r2df\") pod \"payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh\" (UID: \"11e67fc5-e319-4793-9f85-b7d5d3909563\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:56.413556 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.413486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:56.586347 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.586311    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:03:56.683555 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:56.683530    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"]
Sep 11 04:03:56.686426 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:03:56.686374    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e67fc5_e319_4793_9f85_b7d5d3909563.slice/crio-7502d123b4105c8ee7614be03ae7380eb5011e5d55db15506f637f7b2eeb0229 WatchSource:0}: Error finding container 7502d123b4105c8ee7614be03ae7380eb5011e5d55db15506f637f7b2eeb0229: Status 404 returned error can't find the container with id 7502d123b4105c8ee7614be03ae7380eb5011e5d55db15506f637f7b2eeb0229
Sep 11 04:03:57.438426 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:57.438380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh" event={"ID":"11e67fc5-e319-4793-9f85-b7d5d3909563","Type":"ContainerStarted","Data":"b79b7128cb2a8b14a6b5825d9cba7b5c6bf1e398724fa8ae94f6afa205ec187c"}
Sep 11 04:03:57.438867 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:57.438468    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:03:57.438867 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:57.438492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh" event={"ID":"11e67fc5-e319-4793-9f85-b7d5d3909563","Type":"ContainerStarted","Data":"7502d123b4105c8ee7614be03ae7380eb5011e5d55db15506f637f7b2eeb0229"}
Sep 11 04:03:57.455608 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:03:57.455564    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh" podStartSLOduration=1.455546434 podStartE2EDuration="1.455546434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:57.454217516 +0000 UTC m=+835.320237253" watchObservedRunningTime="2026-09-11 04:03:57.455546434 +0000 UTC m=+835.321566173"
Sep 11 04:04:06.578589 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:06.578556    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"]
Sep 11 04:04:08.446305 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:08.446267    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:04:10.484599 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:10.484560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:04:17.013016 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.012982    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"]
Sep 11 04:04:17.013482 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.013214    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f" podUID="71bfae8e-1c6e-47a8-add1-e25ba79726be" containerName="maas-api" containerID="cri-o://d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c" gracePeriod=30
Sep 11 04:04:17.084459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.084401    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"]
Sep 11 04:04:17.084711 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.084684    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9" podUID="c2687d39-dbd8-43ca-8eba-e1b7205c84f2" containerName="payload-pre-processing" containerID="cri-o://b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644" gracePeriod=30
Sep 11 04:04:17.344855 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.344828    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:04:17.357609 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.357579    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:04:17.490972 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.490944    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bfae8e-1c6e-47a8-add1-e25ba79726be-openshift-service-ca\") pod \"71bfae8e-1c6e-47a8-add1-e25ba79726be\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") "
Sep 11 04:04:17.491147 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.491071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-metrics-certs\") pod \"71bfae8e-1c6e-47a8-add1-e25ba79726be\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") "
Sep 11 04:04:17.491147 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.491113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d44x2\" (UniqueName: \"kubernetes.io/projected/71bfae8e-1c6e-47a8-add1-e25ba79726be-kube-api-access-d44x2\") pod \"71bfae8e-1c6e-47a8-add1-e25ba79726be\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") "
Sep 11 04:04:17.491147 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.491134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-maas-api-tls\") pod \"71bfae8e-1c6e-47a8-add1-e25ba79726be\" (UID: \"71bfae8e-1c6e-47a8-add1-e25ba79726be\") "
Sep 11 04:04:17.491317 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.491156    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwc4n\" (UniqueName: \"kubernetes.io/projected/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-kube-api-access-gwc4n\") pod \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\" (UID: \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\") "
Sep 11 04:04:17.491317 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.491229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-plugins-config-volume\") pod \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\" (UID: \"c2687d39-dbd8-43ca-8eba-e1b7205c84f2\") "
Sep 11 04:04:17.491317 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.491268    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bfae8e-1c6e-47a8-add1-e25ba79726be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71bfae8e-1c6e-47a8-add1-e25ba79726be" (UID: "71bfae8e-1c6e-47a8-add1-e25ba79726be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:17.491592 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.491569    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bfae8e-1c6e-47a8-add1-e25ba79726be-openshift-service-ca\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:17.491734 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.491633    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2687d39-dbd8-43ca-8eba-e1b7205c84f2" (UID: "c2687d39-dbd8-43ca-8eba-e1b7205c84f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:17.493343 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.493317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71bfae8e-1c6e-47a8-add1-e25ba79726be" (UID: "71bfae8e-1c6e-47a8-add1-e25ba79726be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:17.493465 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.493354    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71bfae8e-1c6e-47a8-add1-e25ba79726be" (UID: "71bfae8e-1c6e-47a8-add1-e25ba79726be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:17.493615 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.493589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-kube-api-access-gwc4n" (OuterVolumeSpecName: "kube-api-access-gwc4n") pod "c2687d39-dbd8-43ca-8eba-e1b7205c84f2" (UID: "c2687d39-dbd8-43ca-8eba-e1b7205c84f2"). InnerVolumeSpecName "kube-api-access-gwc4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:17.493671 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.493608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bfae8e-1c6e-47a8-add1-e25ba79726be-kube-api-access-d44x2" (OuterVolumeSpecName: "kube-api-access-d44x2") pod "71bfae8e-1c6e-47a8-add1-e25ba79726be" (UID: "71bfae8e-1c6e-47a8-add1-e25ba79726be"). InnerVolumeSpecName "kube-api-access-d44x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:17.537852 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.537779    2566 generic.go:356] "Generic (PLEG): container finished" podID="71bfae8e-1c6e-47a8-add1-e25ba79726be" containerID="d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c" exitCode=0
Sep 11 04:04:17.537852 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.537818    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"
Sep 11 04:04:17.538024 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.537862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f" event={"ID":"71bfae8e-1c6e-47a8-add1-e25ba79726be","Type":"ContainerDied","Data":"d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c"}
Sep 11 04:04:17.538024 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.537916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f" event={"ID":"71bfae8e-1c6e-47a8-add1-e25ba79726be","Type":"ContainerDied","Data":"47a55925d924c70c594005429f512d0a9c050eecabc7a3877f511600867e9e1e"}
Sep 11 04:04:17.538024 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.537941    2566 scope.go:117] "RemoveContainer" containerID="d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c"
Sep 11 04:04:17.539454 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.539430    2566 generic.go:356] "Generic (PLEG): container finished" podID="c2687d39-dbd8-43ca-8eba-e1b7205c84f2" containerID="b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644" exitCode=0
Sep 11 04:04:17.539550 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.539472    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"
Sep 11 04:04:17.539550 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.539500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9" event={"ID":"c2687d39-dbd8-43ca-8eba-e1b7205c84f2","Type":"ContainerDied","Data":"b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644"}
Sep 11 04:04:17.539550 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.539536    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9" event={"ID":"c2687d39-dbd8-43ca-8eba-e1b7205c84f2","Type":"ContainerDied","Data":"86d9c10d2bc4de283a2c3b88443a17361e99d6f874d9b4ba14c9989ac3f6ae57"}
Sep 11 04:04:17.547450 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.547435    2566 scope.go:117] "RemoveContainer" containerID="d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c"
Sep 11 04:04:17.547734 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:04:17.547716    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c\": container with ID starting with d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c not found: ID does not exist" containerID="d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c"
Sep 11 04:04:17.547785 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.547741    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c"} err="failed to get container status \"d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c\": rpc error: code = NotFound desc = could not find container \"d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c\": container with ID starting with d3572a4f073af2db6ad42ace0e1fd2b22dd0d9793ba32ed656342ab1089f8f5c not found: ID does not exist"
Sep 11 04:04:17.547785 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.547757    2566 scope.go:117] "RemoveContainer" containerID="b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644"
Sep 11 04:04:17.555815 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.555799    2566 scope.go:117] "RemoveContainer" containerID="b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644"
Sep 11 04:04:17.556061 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:04:17.556044    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644\": container with ID starting with b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644 not found: ID does not exist" containerID="b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644"
Sep 11 04:04:17.556116 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.556067    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644"} err="failed to get container status \"b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644\": rpc error: code = NotFound desc = could not find container \"b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644\": container with ID starting with b71cbf074e6760a7fcb5322c9fb73689f62fb912ffdbf17c26c29a130931f644 not found: ID does not exist"
Sep 11 04:04:17.563980 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.563958    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"]
Sep 11 04:04:17.567866 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.567846    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-944e93-6785b44586-gxhw9"]
Sep 11 04:04:17.577915 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.577897    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"]
Sep 11 04:04:17.584997 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.584977    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-944e93-7d5bcc9c9f-zq78f"]
Sep 11 04:04:17.592642 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.592622    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:17.592720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.592643    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-metrics-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:17.592720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.592653    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44x2\" (UniqueName: \"kubernetes.io/projected/71bfae8e-1c6e-47a8-add1-e25ba79726be-kube-api-access-d44x2\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:17.592720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.592661    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bfae8e-1c6e-47a8-add1-e25ba79726be-maas-api-tls\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:17.592720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:17.592670    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwc4n\" (UniqueName: \"kubernetes.io/projected/c2687d39-dbd8-43ca-8eba-e1b7205c84f2-kube-api-access-gwc4n\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:18.613186 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:18.613150    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71bfae8e-1c6e-47a8-add1-e25ba79726be" path="/var/lib/kubelet/pods/71bfae8e-1c6e-47a8-add1-e25ba79726be/volumes"
Sep 11 04:04:18.613718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:18.613699    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2687d39-dbd8-43ca-8eba-e1b7205c84f2" path="/var/lib/kubelet/pods/c2687d39-dbd8-43ca-8eba-e1b7205c84f2/volumes"
Sep 11 04:04:21.597565 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:21.597502    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr" podUID="7a6ef017-27d2-4ea1-a5cb-ab84170c686e" containerName="main" containerID="cri-o://568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae" gracePeriod=45
Sep 11 04:04:21.858465 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:21.858394    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:04:22.034071 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034039    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-model-cache\") pod \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") "
Sep 11 04:04:22.034225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034088    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-home\") pod \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") "
Sep 11 04:04:22.034225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034115    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tmp-dir\") pod \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") "
Sep 11 04:04:22.034225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034163    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tls-certs\") pod \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") "
Sep 11 04:04:22.034225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034213    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-dshm\") pod \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") "
Sep 11 04:04:22.034449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034237    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47npv\" (UniqueName: \"kubernetes.io/projected/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-kube-api-access-47npv\") pod \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\" (UID: \"7a6ef017-27d2-4ea1-a5cb-ab84170c686e\") "
Sep 11 04:04:22.034449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034351    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a6ef017-27d2-4ea1-a5cb-ab84170c686e" (UID: "7a6ef017-27d2-4ea1-a5cb-ab84170c686e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:22.034449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034383    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a6ef017-27d2-4ea1-a5cb-ab84170c686e" (UID: "7a6ef017-27d2-4ea1-a5cb-ab84170c686e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:22.034449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034401    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-home" (OuterVolumeSpecName: "home") pod "7a6ef017-27d2-4ea1-a5cb-ab84170c686e" (UID: "7a6ef017-27d2-4ea1-a5cb-ab84170c686e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:22.034647 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034570    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-model-cache\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:22.034647 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034586    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-home\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:22.034647 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.034599    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tmp-dir\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:22.036910 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.036883    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a6ef017-27d2-4ea1-a5cb-ab84170c686e" (UID: "7a6ef017-27d2-4ea1-a5cb-ab84170c686e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:04:22.037028 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.036932    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-dshm" (OuterVolumeSpecName: "dshm") pod "7a6ef017-27d2-4ea1-a5cb-ab84170c686e" (UID: "7a6ef017-27d2-4ea1-a5cb-ab84170c686e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:22.037028 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.036958    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-kube-api-access-47npv" (OuterVolumeSpecName: "kube-api-access-47npv") pod "7a6ef017-27d2-4ea1-a5cb-ab84170c686e" (UID: "7a6ef017-27d2-4ea1-a5cb-ab84170c686e"). InnerVolumeSpecName "kube-api-access-47npv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:22.136025 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.135953    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-dshm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:22.136025 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.135980    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47npv\" (UniqueName: \"kubernetes.io/projected/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-kube-api-access-47npv\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:22.136025 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.135992    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6ef017-27d2-4ea1-a5cb-ab84170c686e-tls-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:22.568768 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.568731    2566 generic.go:356] "Generic (PLEG): container finished" podID="7a6ef017-27d2-4ea1-a5cb-ab84170c686e" containerID="568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae" exitCode=0
Sep 11 04:04:22.568768 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.568769    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"
Sep 11 04:04:22.569021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.568815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr" event={"ID":"7a6ef017-27d2-4ea1-a5cb-ab84170c686e","Type":"ContainerDied","Data":"568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae"}
Sep 11 04:04:22.569021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.568852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr" event={"ID":"7a6ef017-27d2-4ea1-a5cb-ab84170c686e","Type":"ContainerDied","Data":"c0740ea05b9dfba4580c0c41e19811ba7a4707be08bfe4044f0c20e92d7ea0cd"}
Sep 11 04:04:22.569021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.568869    2566 scope.go:117] "RemoveContainer" containerID="568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae"
Sep 11 04:04:22.577646 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.577630    2566 scope.go:117] "RemoveContainer" containerID="568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae"
Sep 11 04:04:22.577936 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:04:22.577918    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae\": container with ID starting with 568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae not found: ID does not exist" containerID="568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae"
Sep 11 04:04:22.577999 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.577948    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae"} err="failed to get container status \"568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae\": rpc error: code = NotFound desc = could not find container \"568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae\": container with ID starting with 568d4b50a1eed990d9ad98b444e26cf1d7938c53d473ebb26ad830c7b62820ae not found: ID does not exist"
Sep 11 04:04:22.589517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.589480    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"]
Sep 11 04:04:22.595374 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.595349    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-944e93/autores-model-944e93-kserve-d848fbff-jjwtr"]
Sep 11 04:04:22.613526 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:22.613489    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6ef017-27d2-4ea1-a5cb-ab84170c686e" path="/var/lib/kubelet/pods/7a6ef017-27d2-4ea1-a5cb-ab84170c686e/volumes"
Sep 11 04:04:24.365308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:24.365273    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"]
Sep 11 04:04:24.365779 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:24.365571    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy" containerID="cri-o://a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d" gracePeriod=30
Sep 11 04:04:24.584681 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:24.584645    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:04:25.078137 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078106    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"]
Sep 11 04:04:25.078524 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078511    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2687d39-dbd8-43ca-8eba-e1b7205c84f2" containerName="payload-pre-processing"
Sep 11 04:04:25.078575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078526    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2687d39-dbd8-43ca-8eba-e1b7205c84f2" containerName="payload-pre-processing"
Sep 11 04:04:25.078575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078537    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71bfae8e-1c6e-47a8-add1-e25ba79726be" containerName="maas-api"
Sep 11 04:04:25.078575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078542    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71bfae8e-1c6e-47a8-add1-e25ba79726be" containerName="maas-api"
Sep 11 04:04:25.078575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078562    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6ef017-27d2-4ea1-a5cb-ab84170c686e" containerName="main"
Sep 11 04:04:25.078575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078567    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6ef017-27d2-4ea1-a5cb-ab84170c686e" containerName="main"
Sep 11 04:04:25.078734 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078621    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6ef017-27d2-4ea1-a5cb-ab84170c686e" containerName="main"
Sep 11 04:04:25.078734 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078631    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2687d39-dbd8-43ca-8eba-e1b7205c84f2" containerName="payload-pre-processing"
Sep 11 04:04:25.078734 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.078638    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71bfae8e-1c6e-47a8-add1-e25ba79726be" containerName="maas-api"
Sep 11 04:04:25.082496 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.082473    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.084876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.084853    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-557c41-openshift-default-dockercfg-pz4wk\""
Sep 11 04:04:25.095847 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.095823    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"]
Sep 11 04:04:25.160285 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b325c4-3194-41c0-9439-7da6a538efa2-istio-podinfo\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.160504 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-data\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.160504 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-certs\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.160673 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-socket\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.160673 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-credential-socket\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.160778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-istio-token\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.160778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b325c4-3194-41c0-9439-7da6a538efa2-istiod-ca-cert\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.160877 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-envoy\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.160877 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.160837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznlp\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-kube-api-access-rznlp\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262008 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.261964    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-socket\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262008 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-credential-socket\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262259 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-istio-token\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262259 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b325c4-3194-41c0-9439-7da6a538efa2-istiod-ca-cert\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262259 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-envoy\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262259 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznlp\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-kube-api-access-rznlp\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262259 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b325c4-3194-41c0-9439-7da6a538efa2-istio-podinfo\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262259 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-data\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262259 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-certs\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262701 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-credential-socket\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262701 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-socket\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262805 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262719    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-data\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.262858 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-certs\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.263022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.262997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b325c4-3194-41c0-9439-7da6a538efa2-istiod-ca-cert\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.264423 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.264383    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-envoy\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.264652 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.264631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b325c4-3194-41c0-9439-7da6a538efa2-istio-podinfo\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.272175 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.272150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-istio-token\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.272532 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.272514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznlp\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-kube-api-access-rznlp\") pod \"e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") " pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.393249 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.393148    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:25.677687 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.677664    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"]
Sep 11 04:04:25.680235 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:04:25.680205    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b325c4_3194_41c0_9439_7da6a538efa2.slice/crio-0a38954bc0e8ab9492dd6fb442f3a4001fc9af46527ce1486447f951dceb5b2e WatchSource:0}: Error finding container 0a38954bc0e8ab9492dd6fb442f3a4001fc9af46527ce1486447f951dceb5b2e: Status 404 returned error can't find the container with id 0a38954bc0e8ab9492dd6fb442f3a4001fc9af46527ce1486447f951dceb5b2e
Sep 11 04:04:25.682952 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.682913    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:04:25.683065 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.682995    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:04:25.683065 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:25.683041    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:04:26.348202 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.348167    2566 patch_prober.go:28] interesting pod/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 04:04:26.348394 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.348230    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 04:04:26.414948 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.414914    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"]
Sep 11 04:04:26.415396 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.415100    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh" podUID="11e67fc5-e319-4793-9f85-b7d5d3909563" containerName="payload-pre-processing" containerID="cri-o://b79b7128cb2a8b14a6b5825d9cba7b5c6bf1e398724fa8ae94f6afa205ec187c" gracePeriod=30
Sep 11 04:04:26.600463 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.600368    2566 generic.go:356] "Generic (PLEG): container finished" podID="11e67fc5-e319-4793-9f85-b7d5d3909563" containerID="b79b7128cb2a8b14a6b5825d9cba7b5c6bf1e398724fa8ae94f6afa205ec187c" exitCode=0
Sep 11 04:04:26.600619 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.600485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh" event={"ID":"11e67fc5-e319-4793-9f85-b7d5d3909563","Type":"ContainerDied","Data":"b79b7128cb2a8b14a6b5825d9cba7b5c6bf1e398724fa8ae94f6afa205ec187c"}
Sep 11 04:04:26.602960 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.602929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" event={"ID":"32b325c4-3194-41c0-9439-7da6a538efa2","Type":"ContainerStarted","Data":"68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b"}
Sep 11 04:04:26.603090 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.602995    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" event={"ID":"32b325c4-3194-41c0-9439-7da6a538efa2","Type":"ContainerStarted","Data":"0a38954bc0e8ab9492dd6fb442f3a4001fc9af46527ce1486447f951dceb5b2e"}
Sep 11 04:04:26.625293 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.625239    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" podStartSLOduration=1.6252204319999999 podStartE2EDuration="1.625220432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:26.624318695 +0000 UTC m=+864.490338473" watchObservedRunningTime="2026-09-11 04:04:26.625220432 +0000 UTC m=+864.491240169"
Sep 11 04:04:26.680769 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.680746    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:04:26.775387 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.775359    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e67fc5-e319-4793-9f85-b7d5d3909563-plugins-config-volume\") pod \"11e67fc5-e319-4793-9f85-b7d5d3909563\" (UID: \"11e67fc5-e319-4793-9f85-b7d5d3909563\") "
Sep 11 04:04:26.775571 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.775487    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r2df\" (UniqueName: \"kubernetes.io/projected/11e67fc5-e319-4793-9f85-b7d5d3909563-kube-api-access-8r2df\") pod \"11e67fc5-e319-4793-9f85-b7d5d3909563\" (UID: \"11e67fc5-e319-4793-9f85-b7d5d3909563\") "
Sep 11 04:04:26.775841 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.775808    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e67fc5-e319-4793-9f85-b7d5d3909563-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11e67fc5-e319-4793-9f85-b7d5d3909563" (UID: "11e67fc5-e319-4793-9f85-b7d5d3909563"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:26.778149 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.778121    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e67fc5-e319-4793-9f85-b7d5d3909563-kube-api-access-8r2df" (OuterVolumeSpecName: "kube-api-access-8r2df") pod "11e67fc5-e319-4793-9f85-b7d5d3909563" (UID: "11e67fc5-e319-4793-9f85-b7d5d3909563"). InnerVolumeSpecName "kube-api-access-8r2df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:26.876488 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.876402    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2df\" (UniqueName: \"kubernetes.io/projected/11e67fc5-e319-4793-9f85-b7d5d3909563-kube-api-access-8r2df\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:26.876488 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:26.876451    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e67fc5-e319-4793-9f85-b7d5d3909563-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:27.393562 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.393532    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:27.399031 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.399008    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:27.609472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.609444    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"
Sep 11 04:04:27.609472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.609462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh" event={"ID":"11e67fc5-e319-4793-9f85-b7d5d3909563","Type":"ContainerDied","Data":"7502d123b4105c8ee7614be03ae7380eb5011e5d55db15506f637f7b2eeb0229"}
Sep 11 04:04:27.609952 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.609539    2566 scope.go:117] "RemoveContainer" containerID="b79b7128cb2a8b14a6b5825d9cba7b5c6bf1e398724fa8ae94f6afa205ec187c"
Sep 11 04:04:27.610015 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.609960    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:27.610876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.610859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:27.652164 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.651520    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"]
Sep 11 04:04:27.654339 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:27.654310    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0e17d64c-b76cfc5f4-24nxh"]
Sep 11 04:04:28.620953 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:28.620915    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e67fc5-e319-4793-9f85-b7d5d3909563" path="/var/lib/kubelet/pods/11e67fc5-e319-4793-9f85-b7d5d3909563/volumes"
Sep 11 04:04:29.618842 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.618815    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:04:29.627866 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.627786    2566 generic.go:356] "Generic (PLEG): container finished" podID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerID="a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d" exitCode=0
Sep 11 04:04:29.627866 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.627850    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"
Sep 11 04:04:29.628314 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.627870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" event={"ID":"b6208c03-d739-42c9-a0e0-2521d4907b94","Type":"ContainerDied","Data":"a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d"}
Sep 11 04:04:29.628314 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.627921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f" event={"ID":"b6208c03-d739-42c9-a0e0-2521d4907b94","Type":"ContainerDied","Data":"97bbcdaee37d27b2080da871fb43a4f9834f47831dec7cd2bfb2421ef09a24c2"}
Sep 11 04:04:29.628314 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.627944    2566 scope.go:117] "RemoveContainer" containerID="a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d"
Sep 11 04:04:29.636295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.636276    2566 scope.go:117] "RemoveContainer" containerID="32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329"
Sep 11 04:04:29.646454 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.646396    2566 scope.go:117] "RemoveContainer" containerID="a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d"
Sep 11 04:04:29.646774 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:04:29.646748    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d\": container with ID starting with a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d not found: ID does not exist" containerID="a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d"
Sep 11 04:04:29.646876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.646784    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d"} err="failed to get container status \"a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d\": rpc error: code = NotFound desc = could not find container \"a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d\": container with ID starting with a6f193977f23177d2b30cb3963db91deac3e0900540911718d74dc933770dd0d not found: ID does not exist"
Sep 11 04:04:29.646876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.646809    2566 scope.go:117] "RemoveContainer" containerID="32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329"
Sep 11 04:04:29.647117 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:04:29.647097    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329\": container with ID starting with 32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329 not found: ID does not exist" containerID="32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329"
Sep 11 04:04:29.647175 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.647121    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329"} err="failed to get container status \"32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329\": rpc error: code = NotFound desc = could not find container \"32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329\": container with ID starting with 32600e9c710a4088c5ce6eeff10757658fd399cece58977396152dc42da54329 not found: ID does not exist"
Sep 11 04:04:29.701615 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701588    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-envoy\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.701788 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701653    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-data\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.701788 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-certs\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.701788 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-token\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.701788 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-podinfo\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.701788 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701778    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxsz4\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-kube-api-access-mxsz4\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.702076 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6208c03-d739-42c9-a0e0-2521d4907b94-istiod-ca-cert\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.702076 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701854    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-credential-socket\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.702076 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.701899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-socket\") pod \"b6208c03-d739-42c9-a0e0-2521d4907b94\" (UID: \"b6208c03-d739-42c9-a0e0-2521d4907b94\") "
Sep 11 04:04:29.702233 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.702198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6208c03-d739-42c9-a0e0-2521d4907b94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:04:29.702295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.702268    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:29.702354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.702314    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6208c03-d739-42c9-a0e0-2521d4907b94-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.703011 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.702633    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:29.703210 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.703176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:29.703359 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.703340    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:29.704994 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.704959    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:04:29.706545 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.706492    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:29.707335 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.707304    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:04:29.707800 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.707756    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-kube-api-access-mxsz4" (OuterVolumeSpecName: "kube-api-access-mxsz4") pod "b6208c03-d739-42c9-a0e0-2521d4907b94" (UID: "b6208c03-d739-42c9-a0e0-2521d4907b94"). InnerVolumeSpecName "kube-api-access-mxsz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:04:29.803575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.803540    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.803575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.803573    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.803755 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.803587    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.803755 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.803600    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.803755 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.803614    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxsz4\" (UniqueName: \"kubernetes.io/projected/b6208c03-d739-42c9-a0e0-2521d4907b94-kube-api-access-mxsz4\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.803755 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.803628    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.803755 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.803643    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.803755 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.803657    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6208c03-d739-42c9-a0e0-2521d4907b94-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:04:29.949985 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.949956    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"]
Sep 11 04:04:29.953533 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:29.953506    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-944e93-openshift-default-777cdf84fd-52k6f"]
Sep 11 04:04:30.407903 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.407864    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"]
Sep 11 04:04:30.408266 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.408251    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy"
Sep 11 04:04:30.408266 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.408267    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy"
Sep 11 04:04:30.408379 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.408284    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e67fc5-e319-4793-9f85-b7d5d3909563" containerName="payload-pre-processing"
Sep 11 04:04:30.408379 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.408290    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e67fc5-e319-4793-9f85-b7d5d3909563" containerName="payload-pre-processing"
Sep 11 04:04:30.408379 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.408369    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy"
Sep 11 04:04:30.408379 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.408380    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e67fc5-e319-4793-9f85-b7d5d3909563" containerName="payload-pre-processing"
Sep 11 04:04:30.412805 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.412789    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:30.415227 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.415207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-557c41-dockercfg-rl4fb\""
Sep 11 04:04:30.415529 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.415507    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-557c41\""
Sep 11 04:04:30.421128 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.421109    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"]
Sep 11 04:04:30.509142 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.509105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swp5z\" (UniqueName: \"kubernetes.io/projected/146f786b-247b-46c8-bbf0-938c3539864c-kube-api-access-swp5z\") pod \"payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj\" (UID: \"146f786b-247b-46c8-bbf0-938c3539864c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:30.509329 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.509296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146f786b-247b-46c8-bbf0-938c3539864c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj\" (UID: \"146f786b-247b-46c8-bbf0-938c3539864c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:30.610073 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.610035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146f786b-247b-46c8-bbf0-938c3539864c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj\" (UID: \"146f786b-247b-46c8-bbf0-938c3539864c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:30.610226 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.610094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swp5z\" (UniqueName: \"kubernetes.io/projected/146f786b-247b-46c8-bbf0-938c3539864c-kube-api-access-swp5z\") pod \"payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj\" (UID: \"146f786b-247b-46c8-bbf0-938c3539864c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:30.610774 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.610754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146f786b-247b-46c8-bbf0-938c3539864c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj\" (UID: \"146f786b-247b-46c8-bbf0-938c3539864c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:30.613114 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.613086    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" path="/var/lib/kubelet/pods/b6208c03-d739-42c9-a0e0-2521d4907b94/volumes"
Sep 11 04:04:30.618220 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.618191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swp5z\" (UniqueName: \"kubernetes.io/projected/146f786b-247b-46c8-bbf0-938c3539864c-kube-api-access-swp5z\") pod \"payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj\" (UID: \"146f786b-247b-46c8-bbf0-938c3539864c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:30.723996 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.723970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:30.994671 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:30.994613    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"]
Sep 11 04:04:30.997317 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:04:30.997287    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146f786b_247b_46c8_bbf0_938c3539864c.slice/crio-b3ed9d5bf7ac96a66a17ddfb5fa5387a24f39630fa4052ddf5e4c56ccfe156ad WatchSource:0}: Error finding container b3ed9d5bf7ac96a66a17ddfb5fa5387a24f39630fa4052ddf5e4c56ccfe156ad: Status 404 returned error can't find the container with id b3ed9d5bf7ac96a66a17ddfb5fa5387a24f39630fa4052ddf5e4c56ccfe156ad
Sep 11 04:04:31.640901 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:31.640867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj" event={"ID":"146f786b-247b-46c8-bbf0-938c3539864c","Type":"ContainerStarted","Data":"dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd"}
Sep 11 04:04:31.641086 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:31.640916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj" event={"ID":"146f786b-247b-46c8-bbf0-938c3539864c","Type":"ContainerStarted","Data":"b3ed9d5bf7ac96a66a17ddfb5fa5387a24f39630fa4052ddf5e4c56ccfe156ad"}
Sep 11 04:04:31.641086 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:31.641017    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:31.658586 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:31.658536    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj" podStartSLOduration=1.6585187540000002 podStartE2EDuration="1.658518754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:31.656533276 +0000 UTC m=+869.522553009" watchObservedRunningTime="2026-09-11 04:04:31.658518754 +0000 UTC m=+869.524538525"
Sep 11 04:04:40.382989 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:40.382957    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:04:42.648776 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:42.648750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:04:42.993856 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:42.993824    2566 patch_prober.go:28] interesting pod/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:50048->10.133.0.51:15021: read: connection reset by peer" start-of-body=
Sep 11 04:04:42.994050 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:42.993893    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:50048->10.133.0.51:15021: read: connection reset by peer"
Sep 11 04:04:43.708871 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:43.708840    2566 generic.go:356] "Generic (PLEG): container finished" podID="32b325c4-3194-41c0-9439-7da6a538efa2" containerID="68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b" exitCode=0
Sep 11 04:04:43.709243 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:43.708917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" event={"ID":"32b325c4-3194-41c0-9439-7da6a538efa2","Type":"ContainerDied","Data":"68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b"}
Sep 11 04:04:43.709306 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:43.709291    2566 scope.go:117] "RemoveContainer" containerID="68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b"
Sep 11 04:04:43.710365 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:43.710331    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:04:43.710512 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:43.710424    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:04:43.710512 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:43.710473    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:04:44.716089 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:44.716056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" event={"ID":"32b325c4-3194-41c0-9439-7da6a538efa2","Type":"ContainerStarted","Data":"e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41"}
Sep 11 04:04:45.393445 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:45.393394    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:45.393696 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:45.393671    2566 patch_prober.go:28] interesting pod/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 04:04:45.393759 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:45.393736    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 04:04:46.192977 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.192932    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"]
Sep 11 04:04:46.193496 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.193361    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy"
Sep 11 04:04:46.193496 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.193375    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy"
Sep 11 04:04:46.193496 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.193462    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6208c03-d739-42c9-a0e0-2521d4907b94" containerName="istio-proxy"
Sep 11 04:04:46.195718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.195702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.199602 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.199584    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d685d\"/\"test-model-5d685d-kserve-self-signed-certs\""
Sep 11 04:04:46.199784 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.199593    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d685d\"/\"default-dockercfg-kfhlq\""
Sep 11 04:04:46.199881 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.199595    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d685d\"/\"openshift-service-ca.crt\""
Sep 11 04:04:46.199938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.199661    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d685d\"/\"kube-root-ca.crt\""
Sep 11 04:04:46.207481 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.207458    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"]
Sep 11 04:04:46.258910 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.258869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b0623f-782c-440f-85be-ee581e820432-tls-certs\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.259183 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.259163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-dshm\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.259314 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.259295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-tmp-dir\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.259459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.259441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-home\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.259594 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.259578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62kq\" (UniqueName: \"kubernetes.io/projected/95b0623f-782c-440f-85be-ee581e820432-kube-api-access-f62kq\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.259771 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.259756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-model-cache\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.361013 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.360970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62kq\" (UniqueName: \"kubernetes.io/projected/95b0623f-782c-440f-85be-ee581e820432-kube-api-access-f62kq\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.361812 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.361782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-model-cache\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.362460 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.362442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b0623f-782c-440f-85be-ee581e820432-tls-certs\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.363205 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.363187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-dshm\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.363705 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.363687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-model-cache\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.364156 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.364086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-tmp-dir\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.364273 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.364113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-tmp-dir\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.364433 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.364401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-home\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.365687 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.364718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-home\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.366341 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.366320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b0623f-782c-440f-85be-ee581e820432-tls-certs\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.367536 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.367516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-dshm\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.373630 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.373602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62kq\" (UniqueName: \"kubernetes.io/projected/95b0623f-782c-440f-85be-ee581e820432-kube-api-access-f62kq\") pod \"test-model-5d685d-kserve-6588b8d5df-8gcjj\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") " pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.394208 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.394187    2566 patch_prober.go:28] interesting pod/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 04:04:46.394363 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.394342    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 04:04:46.509178 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.509150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:46.801905 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:46.801879    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"]
Sep 11 04:04:46.803087 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:04:46.803057    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95b0623f_782c_440f_85be_ee581e820432.slice/crio-4825d45e4ed11caba953fffcae11ba706d0ac19570b06092f2fb65516360355a WatchSource:0}: Error finding container 4825d45e4ed11caba953fffcae11ba706d0ac19570b06092f2fb65516360355a: Status 404 returned error can't find the container with id 4825d45e4ed11caba953fffcae11ba706d0ac19570b06092f2fb65516360355a
Sep 11 04:04:47.393659 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:47.393624    2566 patch_prober.go:28] interesting pod/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 04:04:47.394046 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:47.393679    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 04:04:47.732362 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:47.732330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj" event={"ID":"95b0623f-782c-440f-85be-ee581e820432","Type":"ContainerStarted","Data":"c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228"}
Sep 11 04:04:47.732535 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:47.732369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj" event={"ID":"95b0623f-782c-440f-85be-ee581e820432","Type":"ContainerStarted","Data":"4825d45e4ed11caba953fffcae11ba706d0ac19570b06092f2fb65516360355a"}
Sep 11 04:04:47.753691 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:47.753626    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj" podStartSLOduration=1.7536055 podStartE2EDuration="1.7536055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:04:47.751749473 +0000 UTC m=+885.617769211" watchObservedRunningTime="2026-09-11 04:04:47.7536055 +0000 UTC m=+885.619625238"
Sep 11 04:04:48.393929 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:48.393894    2566 patch_prober.go:28] interesting pod/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 04:04:48.394392 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:48.393958    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 04:04:49.394075 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:49.394040    2566 patch_prober.go:28] interesting pod/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 04:04:49.394469 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:49.394090    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 04:04:50.397111 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:50.397083    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:50.397496 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:50.397435    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:50.398183 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:50.398167    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:04:51.180869 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:51.180833    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:04:56.509573 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:56.509487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:56.509983 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:56.509803    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:56.522086 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:56.522063    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:04:56.786323 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:04:56.786236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:05:02.617070 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:02.617043    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:05:02.623719 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:02.623695    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:05:05.473548 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:05.473517    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:05:20.174385 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.174352    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:05:20.796156 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.796123    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"]
Sep 11 04:05:20.799823 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.799802    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.802127 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.802104    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-557c41\"/\"kube-root-ca.crt\""
Sep 11 04:05:20.802262 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.802132    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-557c41\"/\"explicit-model-557c41-kserve-self-signed-certs\""
Sep 11 04:05:20.802422 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.802388    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-557c41\"/\"openshift-service-ca.crt\""
Sep 11 04:05:20.802489 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.802472    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-557c41\"/\"default-dockercfg-2sm2c\""
Sep 11 04:05:20.809263 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.809243    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"]
Sep 11 04:05:20.870173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.870145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-dshm\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.870298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.870180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-model-cache\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.870298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.870206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2d9\" (UniqueName: \"kubernetes.io/projected/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-kube-api-access-7h2d9\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.870298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.870264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tls-certs\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.870435 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.870365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tmp-dir\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.870475 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.870435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-home\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.971865 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.971833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-home\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.972022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.971883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-dshm\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.972022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.971917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-model-cache\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.972022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.971942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2d9\" (UniqueName: \"kubernetes.io/projected/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-kube-api-access-7h2d9\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.972022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.971968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tls-certs\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.972249 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.972021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tmp-dir\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.972249 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.972200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-home\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.972368 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.972319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-model-cache\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.972458 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.972437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tmp-dir\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.973943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.973922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-dshm\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.974180 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.974162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tls-certs\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:20.979745 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:20.979721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2d9\" (UniqueName: \"kubernetes.io/projected/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-kube-api-access-7h2d9\") pod \"explicit-model-557c41-kserve-74797b4695-ddjgr\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") " pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:21.109526 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:21.109422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:21.389051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:21.389005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"]
Sep 11 04:05:21.391081 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:05:21.391041    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf88d69e0_3ad9_4331_b424_3c073f0e8ce2.slice/crio-c725dabeaa58562f59dd10b22ab51c6859ccf8c383bbaaf7584d7fa8f984d5e3 WatchSource:0}: Error finding container c725dabeaa58562f59dd10b22ab51c6859ccf8c383bbaaf7584d7fa8f984d5e3: Status 404 returned error can't find the container with id c725dabeaa58562f59dd10b22ab51c6859ccf8c383bbaaf7584d7fa8f984d5e3
Sep 11 04:05:21.902461 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:21.902424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr" event={"ID":"f88d69e0-3ad9-4331-b424-3c073f0e8ce2","Type":"ContainerStarted","Data":"190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7"}
Sep 11 04:05:21.902461 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:21.902465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr" event={"ID":"f88d69e0-3ad9-4331-b424-3c073f0e8ce2","Type":"ContainerStarted","Data":"c725dabeaa58562f59dd10b22ab51c6859ccf8c383bbaaf7584d7fa8f984d5e3"}
Sep 11 04:05:21.922459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:21.922389    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr" podStartSLOduration=1.922374738 podStartE2EDuration="1.922374738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:05:21.92153931 +0000 UTC m=+919.787559047" watchObservedRunningTime="2026-09-11 04:05:21.922374738 +0000 UTC m=+919.788394473"
Sep 11 04:05:25.316162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:25.316130    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"]
Sep 11 04:05:31.110278 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:31.110246    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:36.081676 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.081642    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"]
Sep 11 04:05:36.082135 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.081899    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj" podUID="146f786b-247b-46c8-bbf0-938c3539864c" containerName="payload-pre-processing" containerID="cri-o://dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd" gracePeriod=30
Sep 11 04:05:36.329142 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.329114    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:05:36.410995 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.410909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swp5z\" (UniqueName: \"kubernetes.io/projected/146f786b-247b-46c8-bbf0-938c3539864c-kube-api-access-swp5z\") pod \"146f786b-247b-46c8-bbf0-938c3539864c\" (UID: \"146f786b-247b-46c8-bbf0-938c3539864c\") "
Sep 11 04:05:36.411166 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.411028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146f786b-247b-46c8-bbf0-938c3539864c-plugins-config-volume\") pod \"146f786b-247b-46c8-bbf0-938c3539864c\" (UID: \"146f786b-247b-46c8-bbf0-938c3539864c\") "
Sep 11 04:05:36.411375 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.411349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146f786b-247b-46c8-bbf0-938c3539864c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "146f786b-247b-46c8-bbf0-938c3539864c" (UID: "146f786b-247b-46c8-bbf0-938c3539864c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:05:36.413011 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.412989    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146f786b-247b-46c8-bbf0-938c3539864c-kube-api-access-swp5z" (OuterVolumeSpecName: "kube-api-access-swp5z") pod "146f786b-247b-46c8-bbf0-938c3539864c" (UID: "146f786b-247b-46c8-bbf0-938c3539864c"). InnerVolumeSpecName "kube-api-access-swp5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:05:36.511804 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.511766    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146f786b-247b-46c8-bbf0-938c3539864c-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:36.511804 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.511792    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swp5z\" (UniqueName: \"kubernetes.io/projected/146f786b-247b-46c8-bbf0-938c3539864c-kube-api-access-swp5z\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:36.882955 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.882924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:05:36.971319 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.971283    2566 generic.go:356] "Generic (PLEG): container finished" podID="146f786b-247b-46c8-bbf0-938c3539864c" containerID="dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd" exitCode=0
Sep 11 04:05:36.971498 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.971327    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"
Sep 11 04:05:36.971498 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.971366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj" event={"ID":"146f786b-247b-46c8-bbf0-938c3539864c","Type":"ContainerDied","Data":"dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd"}
Sep 11 04:05:36.971498 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.971429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj" event={"ID":"146f786b-247b-46c8-bbf0-938c3539864c","Type":"ContainerDied","Data":"b3ed9d5bf7ac96a66a17ddfb5fa5387a24f39630fa4052ddf5e4c56ccfe156ad"}
Sep 11 04:05:36.971498 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.971457    2566 scope.go:117] "RemoveContainer" containerID="dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd"
Sep 11 04:05:36.980027 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.980009    2566 scope.go:117] "RemoveContainer" containerID="dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd"
Sep 11 04:05:36.980278 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:05:36.980258    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd\": container with ID starting with dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd not found: ID does not exist" containerID="dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd"
Sep 11 04:05:36.980315 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.980287    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd"} err="failed to get container status \"dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd\": rpc error: code = NotFound desc = could not find container \"dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd\": container with ID starting with dc94225b06d61e7354861234d95aff8cefedfcf5c83744280d8a250cdd719fcd not found: ID does not exist"
Sep 11 04:05:36.990706 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.990683    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"]
Sep 11 04:05:36.994244 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:36.994225    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-557c41-5bbb55c74b-t7ddj"]
Sep 11 04:05:38.612569 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:38.612532    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146f786b-247b-46c8-bbf0-938c3539864c" path="/var/lib/kubelet/pods/146f786b-247b-46c8-bbf0-938c3539864c/volumes"
Sep 11 04:05:40.335390 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.335349    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr" podUID="f88d69e0-3ad9-4331-b424-3c073f0e8ce2" containerName="main" containerID="cri-o://190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7" gracePeriod=45
Sep 11 04:05:40.578853 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.578832    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:40.745935 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.745909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-home\") pod \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") "
Sep 11 04:05:40.746083 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.745945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tmp-dir\") pod \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") "
Sep 11 04:05:40.746083 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.745973    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tls-certs\") pod \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") "
Sep 11 04:05:40.746083 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746012    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h2d9\" (UniqueName: \"kubernetes.io/projected/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-kube-api-access-7h2d9\") pod \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") "
Sep 11 04:05:40.746083 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746038    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-model-cache\") pod \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") "
Sep 11 04:05:40.746277 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-dshm\") pod \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\" (UID: \"f88d69e0-3ad9-4331-b424-3c073f0e8ce2\") "
Sep 11 04:05:40.746277 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746190    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-home" (OuterVolumeSpecName: "home") pod "f88d69e0-3ad9-4331-b424-3c073f0e8ce2" (UID: "f88d69e0-3ad9-4331-b424-3c073f0e8ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:40.746277 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746202    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f88d69e0-3ad9-4331-b424-3c073f0e8ce2" (UID: "f88d69e0-3ad9-4331-b424-3c073f0e8ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:40.746580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746303    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f88d69e0-3ad9-4331-b424-3c073f0e8ce2" (UID: "f88d69e0-3ad9-4331-b424-3c073f0e8ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:40.746761 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746743    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-home\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:40.746956 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746933    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tmp-dir\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:40.747016 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.746963    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-model-cache\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:40.748371 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.748342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f88d69e0-3ad9-4331-b424-3c073f0e8ce2" (UID: "f88d69e0-3ad9-4331-b424-3c073f0e8ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:05:40.748513 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.748382    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-dshm" (OuterVolumeSpecName: "dshm") pod "f88d69e0-3ad9-4331-b424-3c073f0e8ce2" (UID: "f88d69e0-3ad9-4331-b424-3c073f0e8ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:40.748513 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.748384    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-kube-api-access-7h2d9" (OuterVolumeSpecName: "kube-api-access-7h2d9") pod "f88d69e0-3ad9-4331-b424-3c073f0e8ce2" (UID: "f88d69e0-3ad9-4331-b424-3c073f0e8ce2"). InnerVolumeSpecName "kube-api-access-7h2d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:05:40.848398 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.848370    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-tls-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:40.848398 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.848393    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2d9\" (UniqueName: \"kubernetes.io/projected/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-kube-api-access-7h2d9\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:40.848569 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.848402    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f88d69e0-3ad9-4331-b424-3c073f0e8ce2-dshm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:40.996917 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.996825    2566 generic.go:356] "Generic (PLEG): container finished" podID="f88d69e0-3ad9-4331-b424-3c073f0e8ce2" containerID="190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7" exitCode=0
Sep 11 04:05:40.996917 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.996873    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"
Sep 11 04:05:40.996917 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.996903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr" event={"ID":"f88d69e0-3ad9-4331-b424-3c073f0e8ce2","Type":"ContainerDied","Data":"190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7"}
Sep 11 04:05:40.997174 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.996946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr" event={"ID":"f88d69e0-3ad9-4331-b424-3c073f0e8ce2","Type":"ContainerDied","Data":"c725dabeaa58562f59dd10b22ab51c6859ccf8c383bbaaf7584d7fa8f984d5e3"}
Sep 11 04:05:40.997174 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:40.996965    2566 scope.go:117] "RemoveContainer" containerID="190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7"
Sep 11 04:05:41.007258 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:41.007243    2566 scope.go:117] "RemoveContainer" containerID="190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7"
Sep 11 04:05:41.007539 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:05:41.007518    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7\": container with ID starting with 190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7 not found: ID does not exist" containerID="190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7"
Sep 11 04:05:41.007599 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:41.007547    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7"} err="failed to get container status \"190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7\": rpc error: code = NotFound desc = could not find container \"190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7\": container with ID starting with 190819c2cff8dc3b6257b75371fa05efe5cfb36b7dd466692e149eba0abf17d7 not found: ID does not exist"
Sep 11 04:05:41.019794 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:41.019773    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"]
Sep 11 04:05:41.023459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:41.023437    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-557c41/explicit-model-557c41-kserve-74797b4695-ddjgr"]
Sep 11 04:05:42.612399 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:42.612369    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f88d69e0-3ad9-4331-b424-3c073f0e8ce2" path="/var/lib/kubelet/pods/f88d69e0-3ad9-4331-b424-3c073f0e8ce2/volumes"
Sep 11 04:05:43.008996 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:43.008964    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"]
Sep 11 04:05:43.009314 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:43.009259    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy" containerID="cri-o://e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41" gracePeriod=30
Sep 11 04:05:45.258159 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:45.258128    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:05:48.252451 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.252426    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:05:48.417593 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417527    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-istio-token\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.417593 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417563    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-data\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.417789 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417603    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b325c4-3194-41c0-9439-7da6a538efa2-istio-podinfo\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.417789 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-certs\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.417789 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-socket\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.417789 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417706    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rznlp\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-kube-api-access-rznlp\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.417789 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-credential-socket\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.418040 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417803    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-envoy\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.418040 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417856    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b325c4-3194-41c0-9439-7da6a538efa2-istiod-ca-cert\") pod \"32b325c4-3194-41c0-9439-7da6a538efa2\" (UID: \"32b325c4-3194-41c0-9439-7da6a538efa2\") "
Sep 11 04:05:48.418040 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:48.418040 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-data" (OuterVolumeSpecName: "istio-data") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:48.418040 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.417990    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:48.418304 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.418163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:48.418304 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.418251    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:48.418304 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.418262    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b325c4-3194-41c0-9439-7da6a538efa2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:05:48.418304 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.418269    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:48.418304 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.418290    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:48.418304 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.418303    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:48.419718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.419693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32b325c4-3194-41c0-9439-7da6a538efa2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:05:48.419853 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.419821    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-kube-api-access-rznlp" (OuterVolumeSpecName: "kube-api-access-rznlp") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "kube-api-access-rznlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:05:48.419957 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.419941    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-istio-token" (OuterVolumeSpecName: "istio-token") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:05:48.420135 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.420117    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32b325c4-3194-41c0-9439-7da6a538efa2" (UID: "32b325c4-3194-41c0-9439-7da6a538efa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:05:48.519092 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.519069    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:48.519092 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.519087    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b325c4-3194-41c0-9439-7da6a538efa2-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:48.519223 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.519097    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rznlp\" (UniqueName: \"kubernetes.io/projected/32b325c4-3194-41c0-9439-7da6a538efa2-kube-api-access-rznlp\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:48.519223 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.519106    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b325c4-3194-41c0-9439-7da6a538efa2-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:48.519223 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:48.519114    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b325c4-3194-41c0-9439-7da6a538efa2-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:05:49.038219 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.038187    2566 generic.go:356] "Generic (PLEG): container finished" podID="32b325c4-3194-41c0-9439-7da6a538efa2" containerID="e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41" exitCode=0
Sep 11 04:05:49.038367 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.038231    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"
Sep 11 04:05:49.038367 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.038272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" event={"ID":"32b325c4-3194-41c0-9439-7da6a538efa2","Type":"ContainerDied","Data":"e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41"}
Sep 11 04:05:49.038367 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.038326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6" event={"ID":"32b325c4-3194-41c0-9439-7da6a538efa2","Type":"ContainerDied","Data":"0a38954bc0e8ab9492dd6fb442f3a4001fc9af46527ce1486447f951dceb5b2e"}
Sep 11 04:05:49.038367 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.038348    2566 scope.go:117] "RemoveContainer" containerID="e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41"
Sep 11 04:05:49.046726 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.046709    2566 scope.go:117] "RemoveContainer" containerID="68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b"
Sep 11 04:05:49.054281 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.054263    2566 scope.go:117] "RemoveContainer" containerID="e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41"
Sep 11 04:05:49.054596 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:05:49.054576    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41\": container with ID starting with e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41 not found: ID does not exist" containerID="e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41"
Sep 11 04:05:49.054669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.054604    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41"} err="failed to get container status \"e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41\": rpc error: code = NotFound desc = could not find container \"e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41\": container with ID starting with e1d3238511b634acda8b35dd15b2b1aced0628ef143d08413f3167fe5c285b41 not found: ID does not exist"
Sep 11 04:05:49.054669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.054620    2566 scope.go:117] "RemoveContainer" containerID="68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b"
Sep 11 04:05:49.054860 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:05:49.054844    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b\": container with ID starting with 68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b not found: ID does not exist" containerID="68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b"
Sep 11 04:05:49.054903 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.054865    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b"} err="failed to get container status \"68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b\": rpc error: code = NotFound desc = could not find container \"68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b\": container with ID starting with 68a4d3c547bc54eb2feafaaa4a84a2171c83f67ad9da9848d156e8dd50b41a4b not found: ID does not exist"
Sep 11 04:05:49.055561 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.055544    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"]
Sep 11 04:05:49.063302 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:49.061252    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-557c41-openshift-default-85fddd4c44-xrfh6"]
Sep 11 04:05:50.612046 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:05:50.612014    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" path="/var/lib/kubelet/pods/32b325c4-3194-41c0-9439-7da6a538efa2/volumes"
Sep 11 04:06:01.860327 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860293    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"]
Sep 11 04:06:01.860821 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860696    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88d69e0-3ad9-4331-b424-3c073f0e8ce2" containerName="main"
Sep 11 04:06:01.860821 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860710    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88d69e0-3ad9-4331-b424-3c073f0e8ce2" containerName="main"
Sep 11 04:06:01.860821 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860724    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy"
Sep 11 04:06:01.860821 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860730    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy"
Sep 11 04:06:01.860821 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860745    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146f786b-247b-46c8-bbf0-938c3539864c" containerName="payload-pre-processing"
Sep 11 04:06:01.860821 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860753    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="146f786b-247b-46c8-bbf0-938c3539864c" containerName="payload-pre-processing"
Sep 11 04:06:01.860821 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860764    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy"
Sep 11 04:06:01.860821 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860770    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy"
Sep 11 04:06:01.861090 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860828    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy"
Sep 11 04:06:01.861090 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860838    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="146f786b-247b-46c8-bbf0-938c3539864c" containerName="payload-pre-processing"
Sep 11 04:06:01.861090 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860843    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b325c4-3194-41c0-9439-7da6a538efa2" containerName="istio-proxy"
Sep 11 04:06:01.861090 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.860848    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88d69e0-3ad9-4331-b424-3c073f0e8ce2" containerName="main"
Sep 11 04:06:01.863778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.863760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.866241 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.866221    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2d69ee02-openshift-default-dockercfg-cvzbx\""
Sep 11 04:06:01.877506 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.877477    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"]
Sep 11 04:06:01.929257 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e59948-d0a0-4c2b-a988-778db687aa48-istio-podinfo\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.929449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-istio-token\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.929449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-data\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.929449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-socket\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.929449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-envoy\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.929621 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk567\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-kube-api-access-lk567\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.929621 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-credential-socket\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.929621 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e59948-d0a0-4c2b-a988-778db687aa48-istiod-ca-cert\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:01.929621 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:01.929572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-certs\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.030856 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.030829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e59948-d0a0-4c2b-a988-778db687aa48-istiod-ca-cert\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.030864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-certs\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.030884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e59948-d0a0-4c2b-a988-778db687aa48-istio-podinfo\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.030913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-istio-token\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.030941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-data\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.030975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-socket\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.031027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-envoy\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031349 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.031051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk567\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-kube-api-access-lk567\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031349 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.031089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-credential-socket\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031349 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.031309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-certs\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031526 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.031442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-credential-socket\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031526 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.031479    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-data\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031657 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.031623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e59948-d0a0-4c2b-a988-778db687aa48-istiod-ca-cert\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.031721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.031694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-socket\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.033828 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.033806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e59948-d0a0-4c2b-a988-778db687aa48-istio-podinfo\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.034522 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.034499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-envoy\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.039116 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.039091    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-istio-token\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.039555 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.039535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk567\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-kube-api-access-lk567\") pod \"e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") " pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.174555 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.174474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:02.446888 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.446867    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"]
Sep 11 04:06:02.449225 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:06:02.449197    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e59948_d0a0_4c2b_a988_778db687aa48.slice/crio-57afd2a91eb733d6495b316c1b63b939b458812eaaaf237e928cebe45ea5df8a WatchSource:0}: Error finding container 57afd2a91eb733d6495b316c1b63b939b458812eaaaf237e928cebe45ea5df8a: Status 404 returned error can't find the container with id 57afd2a91eb733d6495b316c1b63b939b458812eaaaf237e928cebe45ea5df8a
Sep 11 04:06:02.451748 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.451716    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:02.451843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.451779    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:02.451843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:02.451812    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:03.108009 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:03.107929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" event={"ID":"79e59948-d0a0-4c2b-a988-778db687aa48","Type":"ContainerStarted","Data":"1151e6d0ff5b9a9d9ad4424c9b42fb93bfb3cd7439c09906513f5bfb8dd9f73c"}
Sep 11 04:06:03.108009 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:03.107986    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" event={"ID":"79e59948-d0a0-4c2b-a988-778db687aa48","Type":"ContainerStarted","Data":"57afd2a91eb733d6495b316c1b63b939b458812eaaaf237e928cebe45ea5df8a"}
Sep 11 04:06:03.128699 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:03.128654    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" podStartSLOduration=2.128640871 podStartE2EDuration="2.128640871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:06:03.126902039 +0000 UTC m=+960.992921774" watchObservedRunningTime="2026-09-11 04:06:03.128640871 +0000 UTC m=+960.994660605"
Sep 11 04:06:03.175551 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:03.175528    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:03.180389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:03.180369    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:04.112706 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:04.112676    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:04.113498 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:04.113480    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:06.935206 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:06.935170    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"]
Sep 11 04:06:06.940220 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:06.940202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:06.942543 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:06.942521    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2d69ee02\""
Sep 11 04:06:06.942647 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:06.942542    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2d69ee02-dockercfg-4g5vt\""
Sep 11 04:06:06.946955 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:06.946820    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"]
Sep 11 04:06:06.972252 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:06.972230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5pz\" (UniqueName: \"kubernetes.io/projected/21dd4493-6750-4b50-bed7-33413280ddfd-kube-api-access-qj5pz\") pod \"payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc\" (UID: \"21dd4493-6750-4b50-bed7-33413280ddfd\") " pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:06.972363 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:06.972286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd4493-6750-4b50-bed7-33413280ddfd-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc\" (UID: \"21dd4493-6750-4b50-bed7-33413280ddfd\") " pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:07.073209 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:07.073164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd4493-6750-4b50-bed7-33413280ddfd-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc\" (UID: \"21dd4493-6750-4b50-bed7-33413280ddfd\") " pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:07.073395 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:07.073320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5pz\" (UniqueName: \"kubernetes.io/projected/21dd4493-6750-4b50-bed7-33413280ddfd-kube-api-access-qj5pz\") pod \"payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc\" (UID: \"21dd4493-6750-4b50-bed7-33413280ddfd\") " pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:07.073889 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:07.073865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd4493-6750-4b50-bed7-33413280ddfd-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc\" (UID: \"21dd4493-6750-4b50-bed7-33413280ddfd\") " pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:07.080887 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:07.080863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5pz\" (UniqueName: \"kubernetes.io/projected/21dd4493-6750-4b50-bed7-33413280ddfd-kube-api-access-qj5pz\") pod \"payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc\" (UID: \"21dd4493-6750-4b50-bed7-33413280ddfd\") " pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:07.254638 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:07.254608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:07.528967 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:07.528942    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"]
Sep 11 04:06:07.530457 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:06:07.530396    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dd4493_6750_4b50_bed7_33413280ddfd.slice/crio-e04e53348f7813808b5373d22fa45cfe55b2c292ac83f0e9811159b0d8113a83 WatchSource:0}: Error finding container e04e53348f7813808b5373d22fa45cfe55b2c292ac83f0e9811159b0d8113a83: Status 404 returned error can't find the container with id e04e53348f7813808b5373d22fa45cfe55b2c292ac83f0e9811159b0d8113a83
Sep 11 04:06:08.133284 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:08.133243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc" event={"ID":"21dd4493-6750-4b50-bed7-33413280ddfd","Type":"ContainerStarted","Data":"5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938"}
Sep 11 04:06:08.133739 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:08.133294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc" event={"ID":"21dd4493-6750-4b50-bed7-33413280ddfd","Type":"ContainerStarted","Data":"e04e53348f7813808b5373d22fa45cfe55b2c292ac83f0e9811159b0d8113a83"}
Sep 11 04:06:08.133739 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:08.133390    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:08.149773 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:08.149689    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc" podStartSLOduration=2.149674233 podStartE2EDuration="2.149674233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:06:08.148810733 +0000 UTC m=+966.014830468" watchObservedRunningTime="2026-09-11 04:06:08.149674233 +0000 UTC m=+966.015693968"
Sep 11 04:06:09.286445 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:09.286398    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:06:19.140504 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:19.140469    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:19.426886 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:19.426117    2566 patch_prober.go:28] interesting pod/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:37230->10.133.0.55:15021: read: connection reset by peer" start-of-body=
Sep 11 04:06:19.426886 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:19.426193    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:37230->10.133.0.55:15021: read: connection reset by peer"
Sep 11 04:06:20.203675 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:20.203639    2566 generic.go:356] "Generic (PLEG): container finished" podID="79e59948-d0a0-4c2b-a988-778db687aa48" containerID="1151e6d0ff5b9a9d9ad4424c9b42fb93bfb3cd7439c09906513f5bfb8dd9f73c" exitCode=0
Sep 11 04:06:20.204176 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:20.203717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" event={"ID":"79e59948-d0a0-4c2b-a988-778db687aa48","Type":"ContainerDied","Data":"1151e6d0ff5b9a9d9ad4424c9b42fb93bfb3cd7439c09906513f5bfb8dd9f73c"}
Sep 11 04:06:20.204261 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:20.204243    2566 scope.go:117] "RemoveContainer" containerID="1151e6d0ff5b9a9d9ad4424c9b42fb93bfb3cd7439c09906513f5bfb8dd9f73c"
Sep 11 04:06:20.205334 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:20.205310    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:20.205419 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:20.205375    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:20.205469 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:20.205425    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:06:21.210835 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:21.210798    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" event={"ID":"79e59948-d0a0-4c2b-a988-778db687aa48","Type":"ContainerStarted","Data":"0366bc92a4056773de47187fbfff477bdbc3c14933421b0eb807ed4b7957d496"}
Sep 11 04:06:21.994010 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:21.993975    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"]
Sep 11 04:06:22.175495 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:22.175457    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:22.175761 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:22.175740    2566 patch_prober.go:28] interesting pod/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 04:06:22.175816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:22.175786    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 04:06:23.175030 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:23.174999    2566 patch_prober.go:28] interesting pod/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 04:06:23.175451 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:23.175064    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 04:06:24.175473 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:24.175436    2566 patch_prober.go:28] interesting pod/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 04:06:24.175884 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:24.175494    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 04:06:24.188437 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:24.188394    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:06:25.175035 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:25.174951    2566 patch_prober.go:28] interesting pod/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 04:06:25.175035 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:25.175019    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 04:06:26.241249 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:26.241220    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:26.241698 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:26.241541    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:26.243582 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:26.243563    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:33.003140 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.003104    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"]
Sep 11 04:06:33.003715 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.003299    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t" podUID="d820ebb6-2774-4e2f-9858-f03f1974d2f2" containerName="payload-pre-processing" containerID="cri-o://8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189" gracePeriod=30
Sep 11 04:06:33.248985 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.248959    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:06:33.272756 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.272673    2566 generic.go:356] "Generic (PLEG): container finished" podID="d820ebb6-2774-4e2f-9858-f03f1974d2f2" containerID="8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189" exitCode=0
Sep 11 04:06:33.272756 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.272717    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"
Sep 11 04:06:33.272970 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.272754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t" event={"ID":"d820ebb6-2774-4e2f-9858-f03f1974d2f2","Type":"ContainerDied","Data":"8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189"}
Sep 11 04:06:33.272970 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.272802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t" event={"ID":"d820ebb6-2774-4e2f-9858-f03f1974d2f2","Type":"ContainerDied","Data":"a44c061f345c01bd40aafd4a1aea67c4d7dab5d112fdfc73e3beb66f6c7d8f94"}
Sep 11 04:06:33.272970 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.272825    2566 scope.go:117] "RemoveContainer" containerID="8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189"
Sep 11 04:06:33.283106 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.283088    2566 scope.go:117] "RemoveContainer" containerID="8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189"
Sep 11 04:06:33.283455 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:06:33.283396    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189\": container with ID starting with 8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189 not found: ID does not exist" containerID="8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189"
Sep 11 04:06:33.283546 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.283466    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189"} err="failed to get container status \"8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189\": rpc error: code = NotFound desc = could not find container \"8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189\": container with ID starting with 8ce7d9ee5d3258cfb9c7772601d53352b223cf9084f7b2404731167c222e7189 not found: ID does not exist"
Sep 11 04:06:33.424232 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.424193    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w85gg\" (UniqueName: \"kubernetes.io/projected/d820ebb6-2774-4e2f-9858-f03f1974d2f2-kube-api-access-w85gg\") pod \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\" (UID: \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\") "
Sep 11 04:06:33.424452 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.424248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d820ebb6-2774-4e2f-9858-f03f1974d2f2-plugins-config-volume\") pod \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\" (UID: \"d820ebb6-2774-4e2f-9858-f03f1974d2f2\") "
Sep 11 04:06:33.424650 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.424625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d820ebb6-2774-4e2f-9858-f03f1974d2f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d820ebb6-2774-4e2f-9858-f03f1974d2f2" (UID: "d820ebb6-2774-4e2f-9858-f03f1974d2f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:06:33.426312 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.426289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d820ebb6-2774-4e2f-9858-f03f1974d2f2-kube-api-access-w85gg" (OuterVolumeSpecName: "kube-api-access-w85gg") pod "d820ebb6-2774-4e2f-9858-f03f1974d2f2" (UID: "d820ebb6-2774-4e2f-9858-f03f1974d2f2"). InnerVolumeSpecName "kube-api-access-w85gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:33.525785 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.525692    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w85gg\" (UniqueName: \"kubernetes.io/projected/d820ebb6-2774-4e2f-9858-f03f1974d2f2-kube-api-access-w85gg\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:33.525785 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.525732    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d820ebb6-2774-4e2f-9858-f03f1974d2f2-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:33.594731 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.594695    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"]
Sep 11 04:06:33.597886 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:33.597851    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-00a938-7c4889dcdb-hdn9t"]
Sep 11 04:06:34.612787 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:34.612752    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d820ebb6-2774-4e2f-9858-f03f1974d2f2" path="/var/lib/kubelet/pods/d820ebb6-2774-4e2f-9858-f03f1974d2f2/volumes"
Sep 11 04:06:36.881472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:36.881433    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:06:37.012724 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.012653    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj" podUID="95b0623f-782c-440f-85be-ee581e820432" containerName="main" containerID="cri-o://c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228" gracePeriod=45
Sep 11 04:06:37.271462 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.271435    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:06:37.295934 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.295905    2566 generic.go:356] "Generic (PLEG): container finished" podID="95b0623f-782c-440f-85be-ee581e820432" containerID="c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228" exitCode=0
Sep 11 04:06:37.296148 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.295942    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"
Sep 11 04:06:37.296148 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.295988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj" event={"ID":"95b0623f-782c-440f-85be-ee581e820432","Type":"ContainerDied","Data":"c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228"}
Sep 11 04:06:37.296148 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.296032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj" event={"ID":"95b0623f-782c-440f-85be-ee581e820432","Type":"ContainerDied","Data":"4825d45e4ed11caba953fffcae11ba706d0ac19570b06092f2fb65516360355a"}
Sep 11 04:06:37.296148 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.296056    2566 scope.go:117] "RemoveContainer" containerID="c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228"
Sep 11 04:06:37.306331 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.306313    2566 scope.go:117] "RemoveContainer" containerID="c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228"
Sep 11 04:06:37.306610 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:06:37.306591    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228\": container with ID starting with c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228 not found: ID does not exist" containerID="c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228"
Sep 11 04:06:37.306683 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.306618    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228"} err="failed to get container status \"c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228\": rpc error: code = NotFound desc = could not find container \"c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228\": container with ID starting with c0f19293a510e32d0078744954f711a5f481c0e075e76a4111ca0377e44cb228 not found: ID does not exist"
Sep 11 04:06:37.360991 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.360964    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f62kq\" (UniqueName: \"kubernetes.io/projected/95b0623f-782c-440f-85be-ee581e820432-kube-api-access-f62kq\") pod \"95b0623f-782c-440f-85be-ee581e820432\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") "
Sep 11 04:06:37.361125 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.361018    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-model-cache\") pod \"95b0623f-782c-440f-85be-ee581e820432\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") "
Sep 11 04:06:37.361125 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.361059    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-home\") pod \"95b0623f-782c-440f-85be-ee581e820432\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") "
Sep 11 04:06:37.361125 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.361079    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-dshm\") pod \"95b0623f-782c-440f-85be-ee581e820432\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") "
Sep 11 04:06:37.361125 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.361108    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-tmp-dir\") pod \"95b0623f-782c-440f-85be-ee581e820432\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") "
Sep 11 04:06:37.361337 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.361127    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b0623f-782c-440f-85be-ee581e820432-tls-certs\") pod \"95b0623f-782c-440f-85be-ee581e820432\" (UID: \"95b0623f-782c-440f-85be-ee581e820432\") "
Sep 11 04:06:37.361395 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.361329    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-home" (OuterVolumeSpecName: "home") pod "95b0623f-782c-440f-85be-ee581e820432" (UID: "95b0623f-782c-440f-85be-ee581e820432"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:37.361395 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.361347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-model-cache" (OuterVolumeSpecName: "model-cache") pod "95b0623f-782c-440f-85be-ee581e820432" (UID: "95b0623f-782c-440f-85be-ee581e820432"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:37.361505 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.361385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95b0623f-782c-440f-85be-ee581e820432" (UID: "95b0623f-782c-440f-85be-ee581e820432"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:37.363349 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.363324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-dshm" (OuterVolumeSpecName: "dshm") pod "95b0623f-782c-440f-85be-ee581e820432" (UID: "95b0623f-782c-440f-85be-ee581e820432"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:37.363488 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.363355    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b0623f-782c-440f-85be-ee581e820432-kube-api-access-f62kq" (OuterVolumeSpecName: "kube-api-access-f62kq") pod "95b0623f-782c-440f-85be-ee581e820432" (UID: "95b0623f-782c-440f-85be-ee581e820432"). InnerVolumeSpecName "kube-api-access-f62kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:37.363488 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.363362    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b0623f-782c-440f-85be-ee581e820432-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95b0623f-782c-440f-85be-ee581e820432" (UID: "95b0623f-782c-440f-85be-ee581e820432"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:06:37.462440 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.462392    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-dshm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:37.462440 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.462439    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-tmp-dir\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:37.462602 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.462450    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b0623f-782c-440f-85be-ee581e820432-tls-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:37.462602 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.462459    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f62kq\" (UniqueName: \"kubernetes.io/projected/95b0623f-782c-440f-85be-ee581e820432-kube-api-access-f62kq\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:37.462602 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.462468    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-model-cache\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:37.462602 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.462476    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b0623f-782c-440f-85be-ee581e820432-home\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:37.616467 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.616429    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"]
Sep 11 04:06:37.620382 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:37.620352    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5d685d/test-model-5d685d-kserve-6588b8d5df-8gcjj"]
Sep 11 04:06:38.612155 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:38.612120    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b0623f-782c-440f-85be-ee581e820432" path="/var/lib/kubelet/pods/95b0623f-782c-440f-85be-ee581e820432/volumes"
Sep 11 04:06:47.667322 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:47.667285    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"]
Sep 11 04:06:47.667843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:47.667549    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc" podUID="21dd4493-6750-4b50-bed7-33413280ddfd" containerName="payload-processing" containerID="cri-o://5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938" gracePeriod=30
Sep 11 04:06:47.914444 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:47.914418    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:47.956203 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:47.956131    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd4493-6750-4b50-bed7-33413280ddfd-plugins-config-volume\") pod \"21dd4493-6750-4b50-bed7-33413280ddfd\" (UID: \"21dd4493-6750-4b50-bed7-33413280ddfd\") "
Sep 11 04:06:47.956333 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:47.956220    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj5pz\" (UniqueName: \"kubernetes.io/projected/21dd4493-6750-4b50-bed7-33413280ddfd-kube-api-access-qj5pz\") pod \"21dd4493-6750-4b50-bed7-33413280ddfd\" (UID: \"21dd4493-6750-4b50-bed7-33413280ddfd\") "
Sep 11 04:06:47.956623 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:47.956594    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dd4493-6750-4b50-bed7-33413280ddfd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21dd4493-6750-4b50-bed7-33413280ddfd" (UID: "21dd4493-6750-4b50-bed7-33413280ddfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:06:47.958215 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:47.958196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dd4493-6750-4b50-bed7-33413280ddfd-kube-api-access-qj5pz" (OuterVolumeSpecName: "kube-api-access-qj5pz") pod "21dd4493-6750-4b50-bed7-33413280ddfd" (UID: "21dd4493-6750-4b50-bed7-33413280ddfd"). InnerVolumeSpecName "kube-api-access-qj5pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:48.057535 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.057501    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21dd4493-6750-4b50-bed7-33413280ddfd-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:48.057535 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.057527    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5pz\" (UniqueName: \"kubernetes.io/projected/21dd4493-6750-4b50-bed7-33413280ddfd-kube-api-access-qj5pz\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:48.352669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.352635    2566 generic.go:356] "Generic (PLEG): container finished" podID="21dd4493-6750-4b50-bed7-33413280ddfd" containerID="5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938" exitCode=0
Sep 11 04:06:48.352843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.352678    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"
Sep 11 04:06:48.352843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.352718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc" event={"ID":"21dd4493-6750-4b50-bed7-33413280ddfd","Type":"ContainerDied","Data":"5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938"}
Sep 11 04:06:48.352843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.352768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc" event={"ID":"21dd4493-6750-4b50-bed7-33413280ddfd","Type":"ContainerDied","Data":"e04e53348f7813808b5373d22fa45cfe55b2c292ac83f0e9811159b0d8113a83"}
Sep 11 04:06:48.352843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.352790    2566 scope.go:117] "RemoveContainer" containerID="5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938"
Sep 11 04:06:48.361141 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.361126    2566 scope.go:117] "RemoveContainer" containerID="5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938"
Sep 11 04:06:48.361392 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:06:48.361377    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938\": container with ID starting with 5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938 not found: ID does not exist" containerID="5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938"
Sep 11 04:06:48.361444 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.361400    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938"} err="failed to get container status \"5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938\": rpc error: code = NotFound desc = could not find container \"5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938\": container with ID starting with 5daf5318c5dfc3c6f9213deabb10cd994c2dbcb29c532ab12f44d0328e5e9938 not found: ID does not exist"
Sep 11 04:06:48.373455 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.373428    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"]
Sep 11 04:06:48.379838 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.379818    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d69ee02-75547dc497-r5jwc"]
Sep 11 04:06:48.612005 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:48.611927    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21dd4493-6750-4b50-bed7-33413280ddfd" path="/var/lib/kubelet/pods/21dd4493-6750-4b50-bed7-33413280ddfd/volumes"
Sep 11 04:06:50.483907 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:50.483874    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:06:54.288396 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:54.288353    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"]
Sep 11 04:06:54.288927 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:54.288645    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy" containerID="cri-o://0366bc92a4056773de47187fbfff477bdbc3c14933421b0eb807ed4b7957d496" gracePeriod=30
Sep 11 04:06:56.244049 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:56.244001    2566 patch_prober.go:28] interesting pod/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 04:06:56.244475 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:56.244072    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 04:06:56.674359 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:56.674254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:06:57.293200 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:57.293162    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"]
Sep 11 04:06:57.293687 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:57.293479    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy" containerID="cri-o://d24f6666a4577594df0e32b935c47a95695f791af2ffb0a69565eaec9cef0f11" gracePeriod=30
Sep 11 04:06:57.931896 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:57.931857    2566 patch_prober.go:28] interesting pod/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 04:06:57.932073 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:57.931917    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 04:06:59.411578 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.411546    2566 generic.go:356] "Generic (PLEG): container finished" podID="79e59948-d0a0-4c2b-a988-778db687aa48" containerID="0366bc92a4056773de47187fbfff477bdbc3c14933421b0eb807ed4b7957d496" exitCode=0
Sep 11 04:06:59.412019 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.411597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" event={"ID":"79e59948-d0a0-4c2b-a988-778db687aa48","Type":"ContainerDied","Data":"0366bc92a4056773de47187fbfff477bdbc3c14933421b0eb807ed4b7957d496"}
Sep 11 04:06:59.412019 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.411646    2566 scope.go:117] "RemoveContainer" containerID="1151e6d0ff5b9a9d9ad4424c9b42fb93bfb3cd7439c09906513f5bfb8dd9f73c"
Sep 11 04:06:59.541866 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.541834    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:06:59.665147 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665053    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-credential-socket\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665147 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-envoy\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665147 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665145    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-socket\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665188    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e59948-d0a0-4c2b-a988-778db687aa48-istiod-ca-cert\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk567\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-kube-api-access-lk567\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-certs\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665307    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-data\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665339    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-istio-token\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665366    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e59948-d0a0-4c2b-a988-778db687aa48-istio-podinfo\") pod \"79e59948-d0a0-4c2b-a988-778db687aa48\" (UID: \"79e59948-d0a0-4c2b-a988-778db687aa48\") "
Sep 11 04:06:59.665479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:59.665845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665514    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:59.665845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665602    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:59.665845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665629    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e59948-d0a0-4c2b-a988-778db687aa48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:06:59.665845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665743    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:59.665845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665758    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e59948-d0a0-4c2b-a988-778db687aa48-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:59.665845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665775    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:59.665845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665791    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:59.666196 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.665955    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-data" (OuterVolumeSpecName: "istio-data") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:59.667506 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.667482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:06:59.667640 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.667617    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-istio-token" (OuterVolumeSpecName: "istio-token") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:59.667868 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.667840    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-kube-api-access-lk567" (OuterVolumeSpecName: "kube-api-access-lk567") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "kube-api-access-lk567". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:06:59.667972 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.667887    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e59948-d0a0-4c2b-a988-778db687aa48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79e59948-d0a0-4c2b-a988-778db687aa48" (UID: "79e59948-d0a0-4c2b-a988-778db687aa48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:06:59.766594 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.766556    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk567\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-kube-api-access-lk567\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:59.766594 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.766588    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:59.766594 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.766599    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e59948-d0a0-4c2b-a988-778db687aa48-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:59.766834 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.766607    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e59948-d0a0-4c2b-a988-778db687aa48-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:06:59.766834 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:06:59.766614    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e59948-d0a0-4c2b-a988-778db687aa48-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:00.418541 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:00.418510    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"
Sep 11 04:07:00.418995 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:00.418560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2" event={"ID":"79e59948-d0a0-4c2b-a988-778db687aa48","Type":"ContainerDied","Data":"57afd2a91eb733d6495b316c1b63b939b458812eaaaf237e928cebe45ea5df8a"}
Sep 11 04:07:00.418995 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:00.418623    2566 scope.go:117] "RemoveContainer" containerID="0366bc92a4056773de47187fbfff477bdbc3c14933421b0eb807ed4b7957d496"
Sep 11 04:07:00.440270 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:00.440244    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"]
Sep 11 04:07:00.444048 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:00.444022    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2d69ee02-openshift-default-7f9bcdbb7b-fqcx2"]
Sep 11 04:07:00.612059 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:00.612018    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" path="/var/lib/kubelet/pods/79e59948-d0a0-4c2b-a988-778db687aa48/volumes"
Sep 11 04:07:02.431658 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.431624    2566 generic.go:356] "Generic (PLEG): container finished" podID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerID="d24f6666a4577594df0e32b935c47a95695f791af2ffb0a69565eaec9cef0f11" exitCode=0
Sep 11 04:07:02.432112 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.431722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" event={"ID":"188faf8b-3a45-4bf6-8603-a83ac299acd1","Type":"ContainerDied","Data":"d24f6666a4577594df0e32b935c47a95695f791af2ffb0a69565eaec9cef0f11"}
Sep 11 04:07:02.432112 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.431819    2566 scope.go:117] "RemoveContainer" containerID="598ece731d00575b9ba2f482a7b8d641c6c75b1a8c5e6de180996219c2a5b5fa"
Sep 11 04:07:02.550261 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.550232    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:07:02.590669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.590599    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-certs\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.590669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.590651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188faf8b-3a45-4bf6-8603-a83ac299acd1-istiod-ca-cert\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.590880 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.590682    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-envoy\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.590880 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.590717    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz84x\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-kube-api-access-xz84x\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.590880 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.590751    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-socket\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.590880 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.590847    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:02.591099 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.590898    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-credential-socket\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.591099 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.590962    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-podinfo\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.591099 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591001    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-token\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.591099 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:02.591099 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591064    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-data\") pod \"188faf8b-3a45-4bf6-8603-a83ac299acd1\" (UID: \"188faf8b-3a45-4bf6-8603-a83ac299acd1\") "
Sep 11 04:07:02.591347 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591152    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:02.591347 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188faf8b-3a45-4bf6-8603-a83ac299acd1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:07:02.591479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591454    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.591545 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591479    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188faf8b-3a45-4bf6-8603-a83ac299acd1-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.591545 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591494    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.591545 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591507    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.591713 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.591598    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-data" (OuterVolumeSpecName: "istio-data") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:02.593079 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.593047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:07:02.593304 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.593284    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-kube-api-access-xz84x" (OuterVolumeSpecName: "kube-api-access-xz84x") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "kube-api-access-xz84x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:02.593446 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.593427    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:07:02.593644 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.593627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-token" (OuterVolumeSpecName: "istio-token") pod "188faf8b-3a45-4bf6-8603-a83ac299acd1" (UID: "188faf8b-3a45-4bf6-8603-a83ac299acd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:07:02.692113 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.692081    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.692113 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.692111    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.692267 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.692123    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.692267 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.692158    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz84x\" (UniqueName: \"kubernetes.io/projected/188faf8b-3a45-4bf6-8603-a83ac299acd1-kube-api-access-xz84x\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:02.692267 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:02.692172    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188faf8b-3a45-4bf6-8603-a83ac299acd1-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:07:03.439441 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:03.439398    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"
Sep 11 04:07:03.439441 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:03.439434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr" event={"ID":"188faf8b-3a45-4bf6-8603-a83ac299acd1","Type":"ContainerDied","Data":"4eb8e2e546f08dc7c178cb841ea1d2db0797e51530b81f51d695f380bcb97744"}
Sep 11 04:07:03.440102 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:03.439493    2566 scope.go:117] "RemoveContainer" containerID="d24f6666a4577594df0e32b935c47a95695f791af2ffb0a69565eaec9cef0f11"
Sep 11 04:07:03.462697 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:03.462664    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"]
Sep 11 04:07:03.466270 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:03.466245    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5d685d-openshift-default-bcd55c84-hp9tr"]
Sep 11 04:07:04.611960 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:04.611927    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" path="/var/lib/kubelet/pods/188faf8b-3a45-4bf6-8603-a83ac299acd1/volumes"
Sep 11 04:07:06.880022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:06.879988    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:07:07.806538 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.806498    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"]
Sep 11 04:07:07.807102 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807034    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d820ebb6-2774-4e2f-9858-f03f1974d2f2" containerName="payload-pre-processing"
Sep 11 04:07:07.807102 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807105    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d820ebb6-2774-4e2f-9858-f03f1974d2f2" containerName="payload-pre-processing"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807120    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b0623f-782c-440f-85be-ee581e820432" containerName="main"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807129    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b0623f-782c-440f-85be-ee581e820432" containerName="main"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807141    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807150    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807186    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807195    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807208    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21dd4493-6750-4b50-bed7-33413280ddfd" containerName="payload-processing"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807218    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21dd4493-6750-4b50-bed7-33413280ddfd" containerName="payload-processing"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807254    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807263    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807277    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy"
Sep 11 04:07:07.807298 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807285    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy"
Sep 11 04:07:07.807738 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807364    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy"
Sep 11 04:07:07.807738 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807377    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d820ebb6-2774-4e2f-9858-f03f1974d2f2" containerName="payload-pre-processing"
Sep 11 04:07:07.807738 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807387    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21dd4493-6750-4b50-bed7-33413280ddfd" containerName="payload-processing"
Sep 11 04:07:07.807738 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807397    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy"
Sep 11 04:07:07.807738 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.807417    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b0623f-782c-440f-85be-ee581e820432" containerName="main"
Sep 11 04:07:07.812172 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.812153    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.814340 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.814319    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-68ec26-openshift-default-dockercfg-gbnrd\""
Sep 11 04:07:07.830876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.830851    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"]
Sep 11 04:07:07.938043 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrv2r\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-kube-api-access-vrv2r\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.938389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istiod-ca-cert\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.938389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-socket\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.938389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-data\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.938389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-certs\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.938389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-podinfo\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.938389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-token\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.938389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-credential-socket\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:07.938389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:07.938295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-envoy\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039068 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-credential-socket\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039068 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-envoy\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039294 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrv2r\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-kube-api-access-vrv2r\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039294 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istiod-ca-cert\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039294 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-socket\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039294 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-data\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039294 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-certs\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039597 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-podinfo\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039597 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-credential-socket\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039597 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-token\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039817 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-data\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039817 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039676    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-certs\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039931 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-socket\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.039931 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.039903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istiod-ca-cert\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.041446 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.041420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-envoy\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.041875 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.041855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-podinfo\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.047569 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.047541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-token\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.047679 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.047661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrv2r\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-kube-api-access-vrv2r\") pod \"e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") " pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.126006 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.125942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:08.401883 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.401855    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"]
Sep 11 04:07:08.403970 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:07:08.403942    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c9907f_aeab_47fe_83a9_12940ecd7da6.slice/crio-9d51c1ac1aa353d609c0c06cf929b31232958dd5d8dbdb4a11de5d3b85f30d06 WatchSource:0}: Error finding container 9d51c1ac1aa353d609c0c06cf929b31232958dd5d8dbdb4a11de5d3b85f30d06: Status 404 returned error can't find the container with id 9d51c1ac1aa353d609c0c06cf929b31232958dd5d8dbdb4a11de5d3b85f30d06
Sep 11 04:07:08.406485 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.406451    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:08.406588 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.406533    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:08.406588 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.406577    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:08.463885 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:08.463807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" event={"ID":"a2c9907f-aeab-47fe-83a9-12940ecd7da6","Type":"ContainerStarted","Data":"9d51c1ac1aa353d609c0c06cf929b31232958dd5d8dbdb4a11de5d3b85f30d06"}
Sep 11 04:07:09.469205 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:09.469163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" event={"ID":"a2c9907f-aeab-47fe-83a9-12940ecd7da6","Type":"ContainerStarted","Data":"38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766"}
Sep 11 04:07:09.489333 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:09.489287    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" podStartSLOduration=2.489269088 podStartE2EDuration="2.489269088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:09.488284517 +0000 UTC m=+1027.354304253" watchObservedRunningTime="2026-09-11 04:07:09.489269088 +0000 UTC m=+1027.355288825"
Sep 11 04:07:10.126484 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:10.126448    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:10.131702 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:10.131677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:10.474138 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:10.474113    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:10.475459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:10.475434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:13.171244 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.171209    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"]
Sep 11 04:07:13.171645 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.171621    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="188faf8b-3a45-4bf6-8603-a83ac299acd1" containerName="istio-proxy"
Sep 11 04:07:13.171645 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.171635    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e59948-d0a0-4c2b-a988-778db687aa48" containerName="istio-proxy"
Sep 11 04:07:13.174702 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.174681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:13.177551 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.177512    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-68ec26-dockercfg-jfj2q\""
Sep 11 04:07:13.177727 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.177709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-68ec26\""
Sep 11 04:07:13.188448 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.188426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"]
Sep 11 04:07:13.286033 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.285996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4512b40e-1e41-49fc-a92e-e888bbf9dee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb\" (UID: \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:13.286191 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.286047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jff\" (UniqueName: \"kubernetes.io/projected/4512b40e-1e41-49fc-a92e-e888bbf9dee1-kube-api-access-s5jff\") pod \"payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb\" (UID: \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:13.386638 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.386605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4512b40e-1e41-49fc-a92e-e888bbf9dee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb\" (UID: \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:13.386638 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.386641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jff\" (UniqueName: \"kubernetes.io/projected/4512b40e-1e41-49fc-a92e-e888bbf9dee1-kube-api-access-s5jff\") pod \"payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb\" (UID: \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:13.387324 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.387302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4512b40e-1e41-49fc-a92e-e888bbf9dee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb\" (UID: \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:13.394208 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.394184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jff\" (UniqueName: \"kubernetes.io/projected/4512b40e-1e41-49fc-a92e-e888bbf9dee1-kube-api-access-s5jff\") pod \"payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb\" (UID: \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:13.484950 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.484923    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:13.774526 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:13.774505    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"]
Sep 11 04:07:13.776400 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:07:13.776361    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4512b40e_1e41_49fc_a92e_e888bbf9dee1.slice/crio-957d450d1077c2b432616f7700683d6bc1d68884b559bdf8339489c494d2eb1d WatchSource:0}: Error finding container 957d450d1077c2b432616f7700683d6bc1d68884b559bdf8339489c494d2eb1d: Status 404 returned error can't find the container with id 957d450d1077c2b432616f7700683d6bc1d68884b559bdf8339489c494d2eb1d
Sep 11 04:07:14.493035 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:14.493003    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb" event={"ID":"4512b40e-1e41-49fc-a92e-e888bbf9dee1","Type":"ContainerStarted","Data":"e70c521054a1f0e2dc18d745816a7a14cb41bde68bc0795f901919f7768f8375"}
Sep 11 04:07:14.493461 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:14.493058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb" event={"ID":"4512b40e-1e41-49fc-a92e-e888bbf9dee1","Type":"ContainerStarted","Data":"957d450d1077c2b432616f7700683d6bc1d68884b559bdf8339489c494d2eb1d"}
Sep 11 04:07:14.493461 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:14.493131    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:14.510925 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:14.510874    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb" podStartSLOduration=1.510860088 podStartE2EDuration="1.510860088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:14.509237871 +0000 UTC m=+1032.375257609" watchObservedRunningTime="2026-09-11 04:07:14.510860088 +0000 UTC m=+1032.376879823"
Sep 11 04:07:16.177585 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:16.177550    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:07:25.499969 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:25.499936    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:07:26.093253 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.092649    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:48100->10.133.0.57:15021: read: connection reset by peer" start-of-body=
Sep 11 04:07:26.093253 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.092731    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:48100->10.133.0.57:15021: read: connection reset by peer"
Sep 11 04:07:26.553556 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.553520    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerID="38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766" exitCode=0
Sep 11 04:07:26.554022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.553597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" event={"ID":"a2c9907f-aeab-47fe-83a9-12940ecd7da6","Type":"ContainerDied","Data":"38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766"}
Sep 11 04:07:26.554022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.553979    2566 scope.go:117] "RemoveContainer" containerID="38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766"
Sep 11 04:07:26.554985 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.554960    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:26.555075 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.555059    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:26.555136 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.555100    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:26.631281 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.631253    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"]
Sep 11 04:07:26.635520 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.635497    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.638739 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.638718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c158b2-openshift-default-dockercfg-krs8r\""
Sep 11 04:07:26.648163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.648138    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"]
Sep 11 04:07:26.814872 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.814777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66920c5d-f81b-444d-806c-564051d7de1e-istio-podinfo\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.814872 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.814818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-certs\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.814872 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.814854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-socket\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.815167 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.814898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-credential-socket\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.815167 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.814922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-data\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.815167 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.814951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhzf\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-kube-api-access-hxhzf\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.815167 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.814987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-envoy\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.815167 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.815019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-istio-token\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.815167 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.815103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66920c5d-f81b-444d-806c-564051d7de1e-istiod-ca-cert\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916292 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66920c5d-f81b-444d-806c-564051d7de1e-istio-podinfo\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916486 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-certs\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916486 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-socket\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916486 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-credential-socket\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916486 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-data\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916486 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhzf\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-kube-api-access-hxhzf\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916486 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-envoy\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916807 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-istio-token\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916807 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66920c5d-f81b-444d-806c-564051d7de1e-istiod-ca-cert\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916807 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-certs\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916980 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-data\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.916980 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.916891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-credential-socket\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.917089 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.917026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-socket\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.917426 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.917377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66920c5d-f81b-444d-806c-564051d7de1e-istiod-ca-cert\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.918859 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.918834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-envoy\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.918983 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.918965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66920c5d-f81b-444d-806c-564051d7de1e-istio-podinfo\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.925844 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.925821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhzf\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-kube-api-access-hxhzf\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.926075 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.926057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-istio-token\") pod \"e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") " pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:26.948946 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:26.948922    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:27.226607 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.226581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"]
Sep 11 04:07:27.229112 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:07:27.229085    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66920c5d_f81b_444d_806c_564051d7de1e.slice/crio-64324143885eab2aecbfa004ecdb776d590b4c7d8036377ff048d91eb6636751 WatchSource:0}: Error finding container 64324143885eab2aecbfa004ecdb776d590b4c7d8036377ff048d91eb6636751: Status 404 returned error can't find the container with id 64324143885eab2aecbfa004ecdb776d590b4c7d8036377ff048d91eb6636751
Sep 11 04:07:27.231928 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.231892    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:27.232027 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.231975    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:27.232027 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.232016    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:27.568600 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.568500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" event={"ID":"a2c9907f-aeab-47fe-83a9-12940ecd7da6","Type":"ContainerStarted","Data":"5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933"}
Sep 11 04:07:27.571552 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.571519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" event={"ID":"66920c5d-f81b-444d-806c-564051d7de1e","Type":"ContainerStarted","Data":"6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f"}
Sep 11 04:07:27.571697 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.571576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" event={"ID":"66920c5d-f81b-444d-806c-564051d7de1e","Type":"ContainerStarted","Data":"64324143885eab2aecbfa004ecdb776d590b4c7d8036377ff048d91eb6636751"}
Sep 11 04:07:27.618044 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.617994    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podStartSLOduration=1.61797615 podStartE2EDuration="1.61797615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:27.61630285 +0000 UTC m=+1045.482322599" watchObservedRunningTime="2026-09-11 04:07:27.61797615 +0000 UTC m=+1045.483995886"
Sep 11 04:07:27.949272 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:27.949177    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:28.126235 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:28.126207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:28.126449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:28.126392    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 04:07:28.126514 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:28.126480    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 04:07:28.960779 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:28.960540    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:29.126848 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:29.126814    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 04:07:29.127042 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:29.126877    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 04:07:29.580552 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:29.580520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:29.581379 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:29.581352    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:30.126889 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:30.126851    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 04:07:30.127307 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:30.126924    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 04:07:31.126639 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.126608    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 04:07:31.126831 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.126662    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 04:07:31.943362 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.943320    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"]
Sep 11 04:07:31.947448 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.947422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:31.955238 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.955197    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c158b2-dockercfg-x8z75\""
Sep 11 04:07:31.955718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.955612    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c158b2\""
Sep 11 04:07:31.964731 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.964697    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"]
Sep 11 04:07:31.966058 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.966033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cd3324-f1c8-489f-a93d-95695af0a931-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc\" (UID: \"35cd3324-f1c8-489f-a93d-95695af0a931\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:31.966267 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:31.966247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7fm\" (UniqueName: \"kubernetes.io/projected/35cd3324-f1c8-489f-a93d-95695af0a931-kube-api-access-th7fm\") pod \"payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc\" (UID: \"35cd3324-f1c8-489f-a93d-95695af0a931\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:32.068910 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.068868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cd3324-f1c8-489f-a93d-95695af0a931-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc\" (UID: \"35cd3324-f1c8-489f-a93d-95695af0a931\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:32.069126 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.068990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7fm\" (UniqueName: \"kubernetes.io/projected/35cd3324-f1c8-489f-a93d-95695af0a931-kube-api-access-th7fm\") pod \"payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc\" (UID: \"35cd3324-f1c8-489f-a93d-95695af0a931\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:32.069851 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.069819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cd3324-f1c8-489f-a93d-95695af0a931-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc\" (UID: \"35cd3324-f1c8-489f-a93d-95695af0a931\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:32.081644 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.081615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7fm\" (UniqueName: \"kubernetes.io/projected/35cd3324-f1c8-489f-a93d-95695af0a931-kube-api-access-th7fm\") pod \"payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc\" (UID: \"35cd3324-f1c8-489f-a93d-95695af0a931\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:32.126928 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.126902    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 04:07:32.127069 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.126977    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 04:07:32.266109 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.266063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:32.538948 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.538922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"]
Sep 11 04:07:32.541338 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:07:32.541302    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cd3324_f1c8_489f_a93d_95695af0a931.slice/crio-366dadce7cdb66d619e94671e9878193f2d6525a7110635c0ccf208ca6f9f30d WatchSource:0}: Error finding container 366dadce7cdb66d619e94671e9878193f2d6525a7110635c0ccf208ca6f9f30d: Status 404 returned error can't find the container with id 366dadce7cdb66d619e94671e9878193f2d6525a7110635c0ccf208ca6f9f30d
Sep 11 04:07:32.595938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:32.595912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc" event={"ID":"35cd3324-f1c8-489f-a93d-95695af0a931","Type":"ContainerStarted","Data":"366dadce7cdb66d619e94671e9878193f2d6525a7110635c0ccf208ca6f9f30d"}
Sep 11 04:07:33.130564 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:33.130539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:33.130984 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:33.130800    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:33.131524 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:33.131504    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:07:33.602826 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:33.602495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc" event={"ID":"35cd3324-f1c8-489f-a93d-95695af0a931","Type":"ContainerStarted","Data":"476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9"}
Sep 11 04:07:33.602826 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:33.602728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:33.624198 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:33.624150    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc" podStartSLOduration=2.624135672 podStartE2EDuration="2.624135672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:33.622183041 +0000 UTC m=+1051.488202782" watchObservedRunningTime="2026-09-11 04:07:33.624135672 +0000 UTC m=+1051.490155407"
Sep 11 04:07:35.731128 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:35.731092    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:07:36.866362 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.866329    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"]
Sep 11 04:07:36.871072 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.871050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.873945 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.873923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-604323-openshift-default-dockercfg-xxtfp\""
Sep 11 04:07:36.883582 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.883558    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"]
Sep 11 04:07:36.910271 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48989c5-5227-4919-87ad-414b2294e431-istiod-ca-cert\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.910439 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-data\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.910439 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-certs\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.910439 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-credential-socket\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.910439 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48989c5-5227-4919-87ad-414b2294e431-istio-podinfo\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.910661 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-istio-token\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.910661 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9k8b\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-kube-api-access-j9k8b\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.910661 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-socket\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:36.910661 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:36.910532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-envoy\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.011521 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-certs\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.011521 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-credential-socket\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.011806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48989c5-5227-4919-87ad-414b2294e431-istio-podinfo\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.011806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-istio-token\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.011806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9k8b\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-kube-api-access-j9k8b\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.011806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-socket\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.011806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-envoy\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.011806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48989c5-5227-4919-87ad-414b2294e431-istiod-ca-cert\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.012123 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.011977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-credential-socket\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.012123 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.012017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-certs\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.012227 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.012140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-data\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.012227 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.012143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-socket\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.012384 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.012362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-data\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.012587 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.012569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48989c5-5227-4919-87ad-414b2294e431-istiod-ca-cert\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.014012 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.013976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-envoy\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.014263 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.014243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48989c5-5227-4919-87ad-414b2294e431-istio-podinfo\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.019571 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.019542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9k8b\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-kube-api-access-j9k8b\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.019571 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.019549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-istio-token\") pod \"e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") " pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.183163 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.183095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:37.458341 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.458255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"]
Sep 11 04:07:37.468150 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.464811    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:37.468150 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.464917    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:37.468150 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.464957    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:37.625222 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.625189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" event={"ID":"e48989c5-5227-4919-87ad-414b2294e431","Type":"ContainerStarted","Data":"b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374"}
Sep 11 04:07:37.625442 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.625249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" event={"ID":"e48989c5-5227-4919-87ad-414b2294e431","Type":"ContainerStarted","Data":"f51f731cf447c32d8f573fc29e4279efd22e9eee02d0182ac3c98ba2add804fe"}
Sep 11 04:07:37.646326 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:37.646263    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podStartSLOduration=1.646240886 podStartE2EDuration="1.646240886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:37.645890118 +0000 UTC m=+1055.511909855" watchObservedRunningTime="2026-09-11 04:07:37.646240886 +0000 UTC m=+1055.512260624"
Sep 11 04:07:38.184295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:38.184254    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:38.189367 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:38.189341    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:38.631345 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:38.631300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:38.632168 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:38.632147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:44.386207 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:44.386176    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:07:44.613553 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:44.613526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:07:45.623865 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:45.623830    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 04:07:45.624319 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:45.623990    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 04:07:46.679776 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:46.679743    2566 generic.go:356] "Generic (PLEG): container finished" podID="66920c5d-f81b-444d-806c-564051d7de1e" containerID="6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f" exitCode=0
Sep 11 04:07:46.680201 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:46.679820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" event={"ID":"66920c5d-f81b-444d-806c-564051d7de1e","Type":"ContainerDied","Data":"6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f"}
Sep 11 04:07:46.680269 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:46.680250    2566 scope.go:117] "RemoveContainer" containerID="6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f"
Sep 11 04:07:46.681446 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:46.681401    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:46.681515 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:46.681482    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:46.681551 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:46.681516    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:47.687932 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:47.687902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" event={"ID":"66920c5d-f81b-444d-806c-564051d7de1e","Type":"ContainerStarted","Data":"c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340"}
Sep 11 04:07:47.949875 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:47.949772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:47.950061 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:47.949901    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 04:07:47.950061 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:47.949946    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 04:07:48.949918 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:48.949443    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 04:07:48.949918 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:48.949509    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 04:07:49.950365 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:49.950319    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 04:07:49.950869 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:49.950392    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 04:07:50.950142 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:50.950100    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 04:07:50.950339 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:50.950166    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 04:07:51.950173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:51.950129    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 04:07:51.950690 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:51.950196    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 04:07:52.949972 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:52.949927    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 04:07:52.950169 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:52.949997    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 04:07:53.953123 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:53.953090    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:53.953626 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:53.953355    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:53.954097 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:53.954077    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:07:54.673486 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:54.673455    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 04:07:54.673675 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:54.673513    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 04:07:54.729444 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:54.729418    2566 generic.go:356] "Generic (PLEG): container finished" podID="e48989c5-5227-4919-87ad-414b2294e431" containerID="b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374" exitCode=0
Sep 11 04:07:54.729552 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:54.729478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" event={"ID":"e48989c5-5227-4919-87ad-414b2294e431","Type":"ContainerDied","Data":"b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374"}
Sep 11 04:07:54.729962 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:54.729947    2566 scope.go:117] "RemoveContainer" containerID="b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374"
Sep 11 04:07:54.730921 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:54.730900    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:54.730981 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:54.730961    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:54.731021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:54.730990    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:07:55.736144 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:55.736064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" event={"ID":"e48989c5-5227-4919-87ad-414b2294e431","Type":"ContainerStarted","Data":"9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a"}
Sep 11 04:07:56.183954 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:56.183873    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:07:56.184109 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:56.184061    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 11 04:07:56.184195 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:56.184122    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 11 04:07:57.184137 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:57.184094    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 11 04:07:57.184649 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:57.184157    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 11 04:07:58.184219 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:58.184182    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 11 04:07:58.184633 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:58.184242    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 11 04:07:59.184443 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:59.184387    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 11 04:07:59.184828 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:07:59.184465    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 11 04:08:00.183997 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:00.183959    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 11 04:08:00.184183 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:00.184028    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 11 04:08:01.186837 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:01.186805    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:08:01.187245 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:01.187139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:08:01.187700 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:01.187683    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:08:05.214927 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.214896    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7486-608d52"]
Sep 11 04:08:05.217754 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.217735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:05.220733 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.220711    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mhkpw\""
Sep 11 04:08:05.228493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.228466    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-608d52"]
Sep 11 04:08:05.290792 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.290759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65fcq\" (UniqueName: \"kubernetes.io/projected/9ce14754-ae81-45be-befb-59050f58d1a1-kube-api-access-65fcq\") pod \"test-curl-7486-608d52\" (UID: \"9ce14754-ae81-45be-befb-59050f58d1a1\") " pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:05.391669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.391638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65fcq\" (UniqueName: \"kubernetes.io/projected/9ce14754-ae81-45be-befb-59050f58d1a1-kube-api-access-65fcq\") pod \"test-curl-7486-608d52\" (UID: \"9ce14754-ae81-45be-befb-59050f58d1a1\") " pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:05.401048 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.401022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65fcq\" (UniqueName: \"kubernetes.io/projected/9ce14754-ae81-45be-befb-59050f58d1a1-kube-api-access-65fcq\") pod \"test-curl-7486-608d52\" (UID: \"9ce14754-ae81-45be-befb-59050f58d1a1\") " pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:05.527278 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.527209    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:05.793976 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:05.793953    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-608d52"]
Sep 11 04:08:05.796149 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:08:05.796125    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ce14754_ae81_45be_befb_59050f58d1a1.slice/crio-2c3ea6b44ac93c90d724619fc4771a5f8cb6017108f5a8d16fb7bb43c1525c52 WatchSource:0}: Error finding container 2c3ea6b44ac93c90d724619fc4771a5f8cb6017108f5a8d16fb7bb43c1525c52: Status 404 returned error can't find the container with id 2c3ea6b44ac93c90d724619fc4771a5f8cb6017108f5a8d16fb7bb43c1525c52
Sep 11 04:08:06.806456 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:06.806401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-608d52" event={"ID":"9ce14754-ae81-45be-befb-59050f58d1a1","Type":"ContainerStarted","Data":"1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053"}
Sep 11 04:08:06.806456 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:06.806457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-608d52" event={"ID":"9ce14754-ae81-45be-befb-59050f58d1a1","Type":"ContainerStarted","Data":"2c3ea6b44ac93c90d724619fc4771a5f8cb6017108f5a8d16fb7bb43c1525c52"}
Sep 11 04:08:06.823530 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:06.823487    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7486-608d52" podStartSLOduration=1.8234729440000002 podStartE2EDuration="1.823472944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:06.821984436 +0000 UTC m=+1084.688004172" watchObservedRunningTime="2026-09-11 04:08:06.823472944 +0000 UTC m=+1084.689492681"
Sep 11 04:08:07.421806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.421771    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7486-608d52"]
Sep 11 04:08:07.424186 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.424155    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7486-608d52"]
Sep 11 04:08:07.537576 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.537506    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7486-bc5527"]
Sep 11 04:08:07.537892 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.537880    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce14754-ae81-45be-befb-59050f58d1a1" containerName="test-curl-7486-608d52"
Sep 11 04:08:07.537935 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.537894    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce14754-ae81-45be-befb-59050f58d1a1" containerName="test-curl-7486-608d52"
Sep 11 04:08:07.537975 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.537951    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce14754-ae81-45be-befb-59050f58d1a1" containerName="test-curl-7486-608d52"
Sep 11 04:08:07.540127 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.540110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-bc5527"
Sep 11 04:08:07.546089 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.546025    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-bc5527"]
Sep 11 04:08:07.712917 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.712896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnngt\" (UniqueName: \"kubernetes.io/projected/aa76ae84-f708-4498-a64b-9327609a47b0-kube-api-access-dnngt\") pod \"test-curl-7486-bc5527\" (UID: \"aa76ae84-f708-4498-a64b-9327609a47b0\") " pod="opendatahub/test-curl-7486-bc5527"
Sep 11 04:08:07.811496 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:07.811439    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7486-608d52" container="test-curl-7486-608d52"
Sep 11 04:08:07.813711 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.813691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnngt\" (UniqueName: \"kubernetes.io/projected/aa76ae84-f708-4498-a64b-9327609a47b0-kube-api-access-dnngt\") pod \"test-curl-7486-bc5527\" (UID: \"aa76ae84-f708-4498-a64b-9327609a47b0\") " pod="opendatahub/test-curl-7486-bc5527"
Sep 11 04:08:07.821988 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.821969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnngt\" (UniqueName: \"kubernetes.io/projected/aa76ae84-f708-4498-a64b-9327609a47b0-kube-api-access-dnngt\") pod \"test-curl-7486-bc5527\" (UID: \"aa76ae84-f708-4498-a64b-9327609a47b0\") " pod="opendatahub/test-curl-7486-bc5527"
Sep 11 04:08:07.851576 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:07.851551    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-bc5527"
Sep 11 04:08:08.115239 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:08.115216    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-bc5527"]
Sep 11 04:08:08.119686 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:08:08.119641    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa76ae84_f708_4498_a64b_9327609a47b0.slice/crio-2d1eb9f68f28eef9620bc8ef2db972aa039dfd0b693dcc2ec8b39b29734b081a WatchSource:0}: Error finding container 2d1eb9f68f28eef9620bc8ef2db972aa039dfd0b693dcc2ec8b39b29734b081a: Status 404 returned error can't find the container with id 2d1eb9f68f28eef9620bc8ef2db972aa039dfd0b693dcc2ec8b39b29734b081a
Sep 11 04:08:08.817361 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:08.817325    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-bc5527" event={"ID":"aa76ae84-f708-4498-a64b-9327609a47b0","Type":"ContainerStarted","Data":"1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1"}
Sep 11 04:08:08.817756 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:08.817369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-bc5527" event={"ID":"aa76ae84-f708-4498-a64b-9327609a47b0","Type":"ContainerStarted","Data":"2d1eb9f68f28eef9620bc8ef2db972aa039dfd0b693dcc2ec8b39b29734b081a"}
Sep 11 04:08:08.817756 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:08.817482    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7486-608d52" podUID="9ce14754-ae81-45be-befb-59050f58d1a1" containerName="test-curl-7486-608d52" containerID="cri-o://1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053" gracePeriod=2
Sep 11 04:08:08.830082 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:08.830008    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7486-bc5527" podStartSLOduration=1.829994704 podStartE2EDuration="1.829994704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:08.829852676 +0000 UTC m=+1086.695872412" watchObservedRunningTime="2026-09-11 04:08:08.829994704 +0000 UTC m=+1086.696014440"
Sep 11 04:08:09.492543 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.492508    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7486-bc5527"]
Sep 11 04:08:09.496718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.496694    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7486-bc5527"]
Sep 11 04:08:09.607605 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.607536    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7486-9e22fc"]
Sep 11 04:08:09.607922 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.607908    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa76ae84-f708-4498-a64b-9327609a47b0" containerName="test-curl-7486-bc5527"
Sep 11 04:08:09.607972 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.607923    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa76ae84-f708-4498-a64b-9327609a47b0" containerName="test-curl-7486-bc5527"
Sep 11 04:08:09.608007 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.607975    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa76ae84-f708-4498-a64b-9327609a47b0" containerName="test-curl-7486-bc5527"
Sep 11 04:08:09.609699 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.609681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-9e22fc"
Sep 11 04:08:09.619367 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.619340    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-9e22fc"]
Sep 11 04:08:09.731046 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.731019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzpl\" (UniqueName: \"kubernetes.io/projected/205d764e-1de8-40df-b5ef-ff7c66de4e69-kube-api-access-lzzpl\") pod \"test-curl-7486-9e22fc\" (UID: \"205d764e-1de8-40df-b5ef-ff7c66de4e69\") " pod="opendatahub/test-curl-7486-9e22fc"
Sep 11 04:08:09.822454 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:09.822429    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7486-bc5527" container="test-curl-7486-bc5527"
Sep 11 04:08:09.831669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.831641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzpl\" (UniqueName: \"kubernetes.io/projected/205d764e-1de8-40df-b5ef-ff7c66de4e69-kube-api-access-lzzpl\") pod \"test-curl-7486-9e22fc\" (UID: \"205d764e-1de8-40df-b5ef-ff7c66de4e69\") " pod="opendatahub/test-curl-7486-9e22fc"
Sep 11 04:08:09.843073 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.843040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzpl\" (UniqueName: \"kubernetes.io/projected/205d764e-1de8-40df-b5ef-ff7c66de4e69-kube-api-access-lzzpl\") pod \"test-curl-7486-9e22fc\" (UID: \"205d764e-1de8-40df-b5ef-ff7c66de4e69\") " pod="opendatahub/test-curl-7486-9e22fc"
Sep 11 04:08:09.919728 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:09.919670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-9e22fc"
Sep 11 04:08:10.183739 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:10.183716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-9e22fc"]
Sep 11 04:08:10.185824 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:08:10.185798    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod205d764e_1de8_40df_b5ef_ff7c66de4e69.slice/crio-fe564f899166c6f669c36096aa5d96a6397cfa43af62d5868236b6023b4c3366 WatchSource:0}: Error finding container fe564f899166c6f669c36096aa5d96a6397cfa43af62d5868236b6023b4c3366: Status 404 returned error can't find the container with id fe564f899166c6f669c36096aa5d96a6397cfa43af62d5868236b6023b4c3366
Sep 11 04:08:10.829338 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:10.829307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-9e22fc" event={"ID":"205d764e-1de8-40df-b5ef-ff7c66de4e69","Type":"ContainerStarted","Data":"6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f"}
Sep 11 04:08:10.829771 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:10.829358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-9e22fc" event={"ID":"205d764e-1de8-40df-b5ef-ff7c66de4e69","Type":"ContainerStarted","Data":"fe564f899166c6f669c36096aa5d96a6397cfa43af62d5868236b6023b4c3366"}
Sep 11 04:08:10.829771 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:10.829482    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7486-bc5527" podUID="aa76ae84-f708-4498-a64b-9327609a47b0" containerName="test-curl-7486-bc5527" containerID="cri-o://1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1" gracePeriod=2
Sep 11 04:08:10.844586 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:10.844530    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7486-9e22fc" podStartSLOduration=1.844510195 podStartE2EDuration="1.844510195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:10.842197827 +0000 UTC m=+1088.708217564" watchObservedRunningTime="2026-09-11 04:08:10.844510195 +0000 UTC m=+1088.710529930"
Sep 11 04:08:10.968233 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:10.968212    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:11.144495 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.144422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65fcq\" (UniqueName: \"kubernetes.io/projected/9ce14754-ae81-45be-befb-59050f58d1a1-kube-api-access-65fcq\") pod \"9ce14754-ae81-45be-befb-59050f58d1a1\" (UID: \"9ce14754-ae81-45be-befb-59050f58d1a1\") "
Sep 11 04:08:11.146371 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.146338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce14754-ae81-45be-befb-59050f58d1a1-kube-api-access-65fcq" (OuterVolumeSpecName: "kube-api-access-65fcq") pod "9ce14754-ae81-45be-befb-59050f58d1a1" (UID: "9ce14754-ae81-45be-befb-59050f58d1a1"). InnerVolumeSpecName "kube-api-access-65fcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:11.246026 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.246008    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65fcq\" (UniqueName: \"kubernetes.io/projected/9ce14754-ae81-45be-befb-59050f58d1a1-kube-api-access-65fcq\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:08:11.439853 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.439776    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7486-9e22fc"]
Sep 11 04:08:11.444919 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.444881    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7486-9e22fc"]
Sep 11 04:08:11.559847 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.559814    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7486-4e8df0"]
Sep 11 04:08:11.560262 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.560245    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="205d764e-1de8-40df-b5ef-ff7c66de4e69" containerName="test-curl-7486-9e22fc"
Sep 11 04:08:11.560352 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.560265    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="205d764e-1de8-40df-b5ef-ff7c66de4e69" containerName="test-curl-7486-9e22fc"
Sep 11 04:08:11.560443 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.560359    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="205d764e-1de8-40df-b5ef-ff7c66de4e69" containerName="test-curl-7486-9e22fc"
Sep 11 04:08:11.562277 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.562259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-4e8df0"
Sep 11 04:08:11.567575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.567554    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-4e8df0"]
Sep 11 04:08:11.750241 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.750214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5c25\" (UniqueName: \"kubernetes.io/projected/42b03ce6-3dd7-491f-bee1-d96c5b658ffa-kube-api-access-x5c25\") pod \"test-curl-7486-4e8df0\" (UID: \"42b03ce6-3dd7-491f-bee1-d96c5b658ffa\") " pod="opendatahub/test-curl-7486-4e8df0"
Sep 11 04:08:11.835132 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.835104    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ce14754-ae81-45be-befb-59050f58d1a1" containerID="1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053" exitCode=137
Sep 11 04:08:11.835610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.835145    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:11.835610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.835211    2566 scope.go:117] "RemoveContainer" containerID="1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053"
Sep 11 04:08:11.835610 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:11.835531    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7486-9e22fc" container="test-curl-7486-9e22fc"
Sep 11 04:08:11.844092 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.844077    2566 scope.go:117] "RemoveContainer" containerID="1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053"
Sep 11 04:08:11.844345 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:11.844328    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053\": container with ID starting with 1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053 not found: ID does not exist" containerID="1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053"
Sep 11 04:08:11.844429 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.844357    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053"} err="failed to get container status \"1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053\": rpc error: code = NotFound desc = could not find container \"1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053\": container with ID starting with 1ecdbe692cdf0a35368cee39a93fd366af13ea209608a81161d478903f1d9053 not found: ID does not exist"
Sep 11 04:08:11.845527 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:11.845506    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7486-608d52\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="9ce14754-ae81-45be-befb-59050f58d1a1" pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:11.850939 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.850914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5c25\" (UniqueName: \"kubernetes.io/projected/42b03ce6-3dd7-491f-bee1-d96c5b658ffa-kube-api-access-x5c25\") pod \"test-curl-7486-4e8df0\" (UID: \"42b03ce6-3dd7-491f-bee1-d96c5b658ffa\") " pod="opendatahub/test-curl-7486-4e8df0"
Sep 11 04:08:11.858699 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.858677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5c25\" (UniqueName: \"kubernetes.io/projected/42b03ce6-3dd7-491f-bee1-d96c5b658ffa-kube-api-access-x5c25\") pod \"test-curl-7486-4e8df0\" (UID: \"42b03ce6-3dd7-491f-bee1-d96c5b658ffa\") " pod="opendatahub/test-curl-7486-4e8df0"
Sep 11 04:08:11.872584 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:11.872568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-4e8df0"
Sep 11 04:08:12.141354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:12.141332    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-4e8df0"]
Sep 11 04:08:12.142803 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:08:12.142777    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42b03ce6_3dd7_491f_bee1_d96c5b658ffa.slice/crio-0d80f54052edb34616dd8925ebbf06cf792b3759af160e1c722850c35c1d0505 WatchSource:0}: Error finding container 0d80f54052edb34616dd8925ebbf06cf792b3759af160e1c722850c35c1d0505: Status 404 returned error can't find the container with id 0d80f54052edb34616dd8925ebbf06cf792b3759af160e1c722850c35c1d0505
Sep 11 04:08:12.289269 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:12.289234    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:08:12.610538 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:12.610506    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7486-608d52\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="9ce14754-ae81-45be-befb-59050f58d1a1" pod="opendatahub/test-curl-7486-608d52"
Sep 11 04:08:12.612808 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:12.612784    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce14754-ae81-45be-befb-59050f58d1a1" path="/var/lib/kubelet/pods/9ce14754-ae81-45be-befb-59050f58d1a1/volumes"
Sep 11 04:08:12.842853 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:12.842827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-4e8df0" event={"ID":"42b03ce6-3dd7-491f-bee1-d96c5b658ffa","Type":"ContainerStarted","Data":"3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd"}
Sep 11 04:08:12.843248 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:12.842867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-4e8df0" event={"ID":"42b03ce6-3dd7-491f-bee1-d96c5b658ffa","Type":"ContainerStarted","Data":"0d80f54052edb34616dd8925ebbf06cf792b3759af160e1c722850c35c1d0505"}
Sep 11 04:08:12.843248 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:12.843006    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7486-9e22fc" podUID="205d764e-1de8-40df-b5ef-ff7c66de4e69" containerName="test-curl-7486-9e22fc" containerID="cri-o://6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f" gracePeriod=2
Sep 11 04:08:12.856473 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:12.856434    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7486-4e8df0" podStartSLOduration=1.856394548 podStartE2EDuration="1.856394548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:12.856130065 +0000 UTC m=+1090.722149806" watchObservedRunningTime="2026-09-11 04:08:12.856394548 +0000 UTC m=+1090.722414283"
Sep 11 04:08:12.980353 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:12.980328    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-bc5527"
Sep 11 04:08:13.163824 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.163731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnngt\" (UniqueName: \"kubernetes.io/projected/aa76ae84-f708-4498-a64b-9327609a47b0-kube-api-access-dnngt\") pod \"aa76ae84-f708-4498-a64b-9327609a47b0\" (UID: \"aa76ae84-f708-4498-a64b-9327609a47b0\") "
Sep 11 04:08:13.166388 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.166343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa76ae84-f708-4498-a64b-9327609a47b0-kube-api-access-dnngt" (OuterVolumeSpecName: "kube-api-access-dnngt") pod "aa76ae84-f708-4498-a64b-9327609a47b0" (UID: "aa76ae84-f708-4498-a64b-9327609a47b0"). InnerVolumeSpecName "kube-api-access-dnngt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:13.265337 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.265310    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnngt\" (UniqueName: \"kubernetes.io/projected/aa76ae84-f708-4498-a64b-9327609a47b0-kube-api-access-dnngt\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:08:13.440531 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.440448    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7486-4e8df0"]
Sep 11 04:08:13.448916 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.448889    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7486-4e8df0"]
Sep 11 04:08:13.848781 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.848740    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa76ae84-f708-4498-a64b-9327609a47b0" containerID="1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1" exitCode=137
Sep 11 04:08:13.849173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.848791    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-bc5527"
Sep 11 04:08:13.849173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.848846    2566 scope.go:117] "RemoveContainer" containerID="1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1"
Sep 11 04:08:13.849287 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:13.849268    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7486-4e8df0" container="test-curl-7486-4e8df0"
Sep 11 04:08:13.857458 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.857443    2566 scope.go:117] "RemoveContainer" containerID="1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1"
Sep 11 04:08:13.857746 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:13.857729    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1\": container with ID starting with 1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1 not found: ID does not exist" containerID="1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1"
Sep 11 04:08:13.857812 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:13.857754    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1"} err="failed to get container status \"1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1\": rpc error: code = NotFound desc = could not find container \"1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1\": container with ID starting with 1018d8678a187960d679206248323134ece105ce93e55484463444dd2a014ab1 not found: ID does not exist"
Sep 11 04:08:13.859567 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:13.859546    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7486-bc5527\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="aa76ae84-f708-4498-a64b-9327609a47b0" pod="opendatahub/test-curl-7486-bc5527"
Sep 11 04:08:14.613878 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:14.613844    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa76ae84-f708-4498-a64b-9327609a47b0" path="/var/lib/kubelet/pods/aa76ae84-f708-4498-a64b-9327609a47b0/volumes"
Sep 11 04:08:14.854434 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:14.854389    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7486-4e8df0" podUID="42b03ce6-3dd7-491f-bee1-d96c5b658ffa" containerName="test-curl-7486-4e8df0" containerID="cri-o://3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd" gracePeriod=2
Sep 11 04:08:15.012400 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.012376    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-9e22fc"
Sep 11 04:08:15.182964 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.182889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzzpl\" (UniqueName: \"kubernetes.io/projected/205d764e-1de8-40df-b5ef-ff7c66de4e69-kube-api-access-lzzpl\") pod \"205d764e-1de8-40df-b5ef-ff7c66de4e69\" (UID: \"205d764e-1de8-40df-b5ef-ff7c66de4e69\") "
Sep 11 04:08:15.184842 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.184814    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205d764e-1de8-40df-b5ef-ff7c66de4e69-kube-api-access-lzzpl" (OuterVolumeSpecName: "kube-api-access-lzzpl") pod "205d764e-1de8-40df-b5ef-ff7c66de4e69" (UID: "205d764e-1de8-40df-b5ef-ff7c66de4e69"). InnerVolumeSpecName "kube-api-access-lzzpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:15.284641 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.284602    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzpl\" (UniqueName: \"kubernetes.io/projected/205d764e-1de8-40df-b5ef-ff7c66de4e69-kube-api-access-lzzpl\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:08:15.862043 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.861957    2566 generic.go:356] "Generic (PLEG): container finished" podID="205d764e-1de8-40df-b5ef-ff7c66de4e69" containerID="6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f" exitCode=137
Sep 11 04:08:15.862043 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.862044    2566 scope.go:117] "RemoveContainer" containerID="6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f"
Sep 11 04:08:15.862726 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.862070    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-9e22fc"
Sep 11 04:08:15.872523 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.872507    2566 scope.go:117] "RemoveContainer" containerID="6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f"
Sep 11 04:08:15.872771 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:15.872753    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f\": container with ID starting with 6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f not found: ID does not exist" containerID="6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f"
Sep 11 04:08:15.872831 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:15.872781    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f"} err="failed to get container status \"6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f\": rpc error: code = NotFound desc = could not find container \"6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f\": container with ID starting with 6d92cf5d9556aa640795938dccc9ed1d12f404d8202e3389f7b1b5cd8c76f73f not found: ID does not exist"
Sep 11 04:08:15.874152 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:15.874132    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7486-9e22fc\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="205d764e-1de8-40df-b5ef-ff7c66de4e69" pod="opendatahub/test-curl-7486-9e22fc"
Sep 11 04:08:16.612236 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:16.612195    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="205d764e-1de8-40df-b5ef-ff7c66de4e69" path="/var/lib/kubelet/pods/205d764e-1de8-40df-b5ef-ff7c66de4e69/volumes"
Sep 11 04:08:17.024993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.024970    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-4e8df0"
Sep 11 04:08:17.097029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.097002    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5c25\" (UniqueName: \"kubernetes.io/projected/42b03ce6-3dd7-491f-bee1-d96c5b658ffa-kube-api-access-x5c25\") pod \"42b03ce6-3dd7-491f-bee1-d96c5b658ffa\" (UID: \"42b03ce6-3dd7-491f-bee1-d96c5b658ffa\") "
Sep 11 04:08:17.098918 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.098891    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b03ce6-3dd7-491f-bee1-d96c5b658ffa-kube-api-access-x5c25" (OuterVolumeSpecName: "kube-api-access-x5c25") pod "42b03ce6-3dd7-491f-bee1-d96c5b658ffa" (UID: "42b03ce6-3dd7-491f-bee1-d96c5b658ffa"). InnerVolumeSpecName "kube-api-access-x5c25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:08:17.197745 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.197689    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5c25\" (UniqueName: \"kubernetes.io/projected/42b03ce6-3dd7-491f-bee1-d96c5b658ffa-kube-api-access-x5c25\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:08:17.874255 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.874220    2566 generic.go:356] "Generic (PLEG): container finished" podID="42b03ce6-3dd7-491f-bee1-d96c5b658ffa" containerID="3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd" exitCode=137
Sep 11 04:08:17.874442 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.874260    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-4e8df0"
Sep 11 04:08:17.874442 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.874316    2566 scope.go:117] "RemoveContainer" containerID="3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd"
Sep 11 04:08:17.882669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.882651    2566 scope.go:117] "RemoveContainer" containerID="3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd"
Sep 11 04:08:17.882950 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:17.882932    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd\": container with ID starting with 3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd not found: ID does not exist" containerID="3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd"
Sep 11 04:08:17.882989 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:17.882968    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd"} err="failed to get container status \"3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd\": rpc error: code = NotFound desc = could not find container \"3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd\": container with ID starting with 3d61cada49eecdc6513875a3058e1a3b1772ba59ace905ef102edc160bb50dfd not found: ID does not exist"
Sep 11 04:08:17.884916 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:08:17.884893    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7486-4e8df0\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="42b03ce6-3dd7-491f-bee1-d96c5b658ffa" pod="opendatahub/test-curl-7486-4e8df0"
Sep 11 04:08:18.612528 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:18.612495    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b03ce6-3dd7-491f-bee1-d96c5b658ffa" path="/var/lib/kubelet/pods/42b03ce6-3dd7-491f-bee1-d96c5b658ffa/volumes"
Sep 11 04:08:19.812740 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.812712    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"]
Sep 11 04:08:19.813121 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.813109    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b03ce6-3dd7-491f-bee1-d96c5b658ffa" containerName="test-curl-7486-4e8df0"
Sep 11 04:08:19.813165 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.813123    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b03ce6-3dd7-491f-bee1-d96c5b658ffa" containerName="test-curl-7486-4e8df0"
Sep 11 04:08:19.813204 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.813197    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b03ce6-3dd7-491f-bee1-d96c5b658ffa" containerName="test-curl-7486-4e8df0"
Sep 11 04:08:19.817642 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.817624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:19.821749 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.821728    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0347a9fa\""
Sep 11 04:08:19.821968 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.821946    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0347a9fa-dockercfg-jf29s\""
Sep 11 04:08:19.835688 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.835664    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"]
Sep 11 04:08:19.860428 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.858182    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"]
Sep 11 04:08:19.880780 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.880752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:19.885790 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.885763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"]
Sep 11 04:08:19.918430 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.918380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jdd\" (UniqueName: \"kubernetes.io/projected/8cb39139-5436-495b-bb0c-7014f4a7ea4c-kube-api-access-k5jdd\") pod \"payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx\" (UID: \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\") " pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:19.918547 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.918443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4x7\" (UniqueName: \"kubernetes.io/projected/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-kube-api-access-pr4x7\") pod \"payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw\" (UID: \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:19.918547 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.918478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb39139-5436-495b-bb0c-7014f4a7ea4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx\" (UID: \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\") " pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:19.918547 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:19.918510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw\" (UID: \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:20.019208 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.019177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw\" (UID: \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:20.019357 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.019232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jdd\" (UniqueName: \"kubernetes.io/projected/8cb39139-5436-495b-bb0c-7014f4a7ea4c-kube-api-access-k5jdd\") pod \"payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx\" (UID: \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\") " pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:20.019357 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.019274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4x7\" (UniqueName: \"kubernetes.io/projected/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-kube-api-access-pr4x7\") pod \"payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw\" (UID: \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:20.019357 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.019293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb39139-5436-495b-bb0c-7014f4a7ea4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx\" (UID: \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\") " pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:20.019968 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.019945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb39139-5436-495b-bb0c-7014f4a7ea4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx\" (UID: \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\") " pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:20.020052 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.019945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw\" (UID: \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:20.029097 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.029077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4x7\" (UniqueName: \"kubernetes.io/projected/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-kube-api-access-pr4x7\") pod \"payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw\" (UID: \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:20.029222 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.029203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jdd\" (UniqueName: \"kubernetes.io/projected/8cb39139-5436-495b-bb0c-7014f4a7ea4c-kube-api-access-k5jdd\") pod \"payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx\" (UID: \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\") " pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:20.127309 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.127234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:20.192909 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.192878    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:20.402402 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.402374    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"]
Sep 11 04:08:20.404443 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:08:20.404402    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ba8e79_fa6e_420f_b561_e1d6a9d93ef7.slice/crio-f453e3b85e17108c75f34aebb3f17ac7b3e3a3d2877e8983a0cdb2287394d5ca WatchSource:0}: Error finding container f453e3b85e17108c75f34aebb3f17ac7b3e3a3d2877e8983a0cdb2287394d5ca: Status 404 returned error can't find the container with id f453e3b85e17108c75f34aebb3f17ac7b3e3a3d2877e8983a0cdb2287394d5ca
Sep 11 04:08:20.479445 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.479389    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"]
Sep 11 04:08:20.483849 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:08:20.483822    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb39139_5436_495b_bb0c_7014f4a7ea4c.slice/crio-b1d58ecb2702320582af276ffb6709acf5a3699470e3df1a77d8d48f58e7d9b2 WatchSource:0}: Error finding container b1d58ecb2702320582af276ffb6709acf5a3699470e3df1a77d8d48f58e7d9b2: Status 404 returned error can't find the container with id b1d58ecb2702320582af276ffb6709acf5a3699470e3df1a77d8d48f58e7d9b2
Sep 11 04:08:20.894422 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.894369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx" event={"ID":"8cb39139-5436-495b-bb0c-7014f4a7ea4c","Type":"ContainerStarted","Data":"f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b"}
Sep 11 04:08:20.894875 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.894486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx" event={"ID":"8cb39139-5436-495b-bb0c-7014f4a7ea4c","Type":"ContainerStarted","Data":"b1d58ecb2702320582af276ffb6709acf5a3699470e3df1a77d8d48f58e7d9b2"}
Sep 11 04:08:20.894875 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.894521    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:20.896107 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.896083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw" event={"ID":"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7","Type":"ContainerStarted","Data":"2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b"}
Sep 11 04:08:20.896225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.896134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:20.896225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.896152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw" event={"ID":"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7","Type":"ContainerStarted","Data":"f453e3b85e17108c75f34aebb3f17ac7b3e3a3d2877e8983a0cdb2287394d5ca"}
Sep 11 04:08:20.912568 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.912521    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx" podStartSLOduration=1.912502984 podStartE2EDuration="1.912502984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:20.911694631 +0000 UTC m=+1098.777714366" watchObservedRunningTime="2026-09-11 04:08:20.912502984 +0000 UTC m=+1098.778522745"
Sep 11 04:08:20.930184 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:20.930142    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw" podStartSLOduration=1.930129667 podStartE2EDuration="1.930129667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:20.92795204 +0000 UTC m=+1098.793971776" watchObservedRunningTime="2026-09-11 04:08:20.930129667 +0000 UTC m=+1098.796149404"
Sep 11 04:08:24.596228 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.596189    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"]
Sep 11 04:08:24.599898 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.599881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.602726 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.602702    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"kube-root-ca.crt\""
Sep 11 04:08:24.602858 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.602704    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"openshift-service-ca.crt\""
Sep 11 04:08:24.604056 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.604035    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"default-dockercfg-ps8cb\""
Sep 11 04:08:24.604155 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.604124    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c158b2\"/\"auth-test-model-c158b2-kserve-self-signed-certs\""
Sep 11 04:08:24.624875 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.624849    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"]
Sep 11 04:08:24.660637 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.660609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-model-cache\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.660768 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.660674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb9vs\" (UniqueName: \"kubernetes.io/projected/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-kube-api-access-sb9vs\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.660768 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.660711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tls-certs\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.660918 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.660825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-home\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.660918 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.660873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-dshm\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.661046 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.660920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tmp-dir\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.761623 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.761594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb9vs\" (UniqueName: \"kubernetes.io/projected/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-kube-api-access-sb9vs\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.761757 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.761629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tls-certs\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.761757 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.761670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-home\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.761757 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.761697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-dshm\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.761757 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.761727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tmp-dir\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.761963 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.761870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-model-cache\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.762099 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.762077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tmp-dir\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.762165 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.762111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-home\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.762209 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.762185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-model-cache\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.764045 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.764013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-dshm\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.764220 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.764201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tls-certs\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.770017 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.769995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb9vs\" (UniqueName: \"kubernetes.io/projected/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-kube-api-access-sb9vs\") pod \"auth-test-model-c158b2-kserve-76559d864c-sjprv\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") " pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:24.911467 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:24.911360    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:25.186001 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:25.185979    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"]
Sep 11 04:08:25.187738 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:08:25.187713    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73ab450d_e9f2_4cfb_8dc8_0e964e6a2052.slice/crio-25a07f165d501ddb4f4c4769963345a0f9992d6cca6a7e2ec52a425a849e310e WatchSource:0}: Error finding container 25a07f165d501ddb4f4c4769963345a0f9992d6cca6a7e2ec52a425a849e310e: Status 404 returned error can't find the container with id 25a07f165d501ddb4f4c4769963345a0f9992d6cca6a7e2ec52a425a849e310e
Sep 11 04:08:25.923533 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:25.923497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv" event={"ID":"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052","Type":"ContainerStarted","Data":"f09719ad226a35ce01d4c6c53e549aa58521c9bf04026423aee01cd69bb7689d"}
Sep 11 04:08:25.923533 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:25.923538    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv" event={"ID":"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052","Type":"ContainerStarted","Data":"25a07f165d501ddb4f4c4769963345a0f9992d6cca6a7e2ec52a425a849e310e"}
Sep 11 04:08:25.939808 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:25.939757    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv" podStartSLOduration=1.939742167 podStartE2EDuration="1.939742167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:25.939178739 +0000 UTC m=+1103.805198489" watchObservedRunningTime="2026-09-11 04:08:25.939742167 +0000 UTC m=+1103.805761902"
Sep 11 04:08:28.583576 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:28.583546    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:08:31.902932 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:31.902902    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:08:31.903430 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:31.902981    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:08:34.911996 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:34.911963    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:34.912441 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:34.912014    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:34.924206 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:34.924179    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:34.985735 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:34.985712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:08:44.384172 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:44.384141    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:08:54.005037 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.005003    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"]
Sep 11 04:08:54.009710 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.009688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.014299 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.014270    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0347a9fa\"/\"e2e-lifecycle-model-0347a9fa-kserve-self-signed-certs\""
Sep 11 04:08:54.014299 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.014293    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0347a9fa\"/\"kube-root-ca.crt\""
Sep 11 04:08:54.014488 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.014315    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0347a9fa\"/\"openshift-service-ca.crt\""
Sep 11 04:08:54.014488 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.014396    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0347a9fa\"/\"default-dockercfg-pgsx5\""
Sep 11 04:08:54.020833 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.020814    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"]
Sep 11 04:08:54.116633 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.116605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-home\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.116778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.116646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-model-cache\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.116778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.116703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrt8f\" (UniqueName: \"kubernetes.io/projected/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-kube-api-access-vrt8f\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.116778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.116742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tls-certs\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.116778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.116769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tmp-dir\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.116993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.116860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-dshm\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218119 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrt8f\" (UniqueName: \"kubernetes.io/projected/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-kube-api-access-vrt8f\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218330 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tls-certs\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218330 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tmp-dir\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218330 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-dshm\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218536 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-home\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218536 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-model-cache\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218724 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-home\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218794 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tmp-dir\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.218858 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.218827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-model-cache\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.220460 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.220441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-dshm\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.220753 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.220739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tls-certs\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.227425 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.227384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrt8f\" (UniqueName: \"kubernetes.io/projected/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-kube-api-access-vrt8f\") pod \"e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") " pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.319665 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.319595    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:08:54.603057 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:54.603025    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"]
Sep 11 04:08:54.605663 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:08:54.605632    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96bde8f6_b372_4df2_9336_cc7f61e7ddd0.slice/crio-c70725a936151e6f57ef80469cdbba1cb50e2aaae820ca8cd585452b95c5db55 WatchSource:0}: Error finding container c70725a936151e6f57ef80469cdbba1cb50e2aaae820ca8cd585452b95c5db55: Status 404 returned error can't find the container with id c70725a936151e6f57ef80469cdbba1cb50e2aaae820ca8cd585452b95c5db55
Sep 11 04:08:55.078346 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:55.078309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd" event={"ID":"96bde8f6-b372-4df2-9336-cc7f61e7ddd0","Type":"ContainerStarted","Data":"1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348"}
Sep 11 04:08:55.078346 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:55.078352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd" event={"ID":"96bde8f6-b372-4df2-9336-cc7f61e7ddd0","Type":"ContainerStarted","Data":"c70725a936151e6f57ef80469cdbba1cb50e2aaae820ca8cd585452b95c5db55"}
Sep 11 04:08:55.099970 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:08:55.099928    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd" podStartSLOduration=2.099912436 podStartE2EDuration="2.099912436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:55.097690119 +0000 UTC m=+1132.963709857" watchObservedRunningTime="2026-09-11 04:08:55.099912436 +0000 UTC m=+1132.965932172"
Sep 11 04:09:01.612916 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:01.612884    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:09:04.320974 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:04.320948    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:09:04.321349 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:04.320988    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:09:04.333463 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:04.333440    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:09:05.141347 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:05.141319    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:09:18.289449 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:18.289400    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:09:36.689331 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:36.689292    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:09:37.499913 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.499885    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"]
Sep 11 04:09:37.500115 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.500092    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx" podUID="8cb39139-5436-495b-bb0c-7014f4a7ea4c" containerName="payload-processing" containerID="cri-o://f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b" gracePeriod=30
Sep 11 04:09:37.510055 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.510028    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"]
Sep 11 04:09:37.510299 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.510257    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw" podUID="e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7" containerName="payload-pre-processing" containerID="cri-o://2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b" gracePeriod=30
Sep 11 04:09:37.781326 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.781305    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:09:37.800002 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.799982    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:09:37.902377 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.902347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-plugins-config-volume\") pod \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\" (UID: \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\") "
Sep 11 04:09:37.902580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.902429    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5jdd\" (UniqueName: \"kubernetes.io/projected/8cb39139-5436-495b-bb0c-7014f4a7ea4c-kube-api-access-k5jdd\") pod \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\" (UID: \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\") "
Sep 11 04:09:37.902580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.902464    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb39139-5436-495b-bb0c-7014f4a7ea4c-plugins-config-volume\") pod \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\" (UID: \"8cb39139-5436-495b-bb0c-7014f4a7ea4c\") "
Sep 11 04:09:37.902580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.902482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr4x7\" (UniqueName: \"kubernetes.io/projected/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-kube-api-access-pr4x7\") pod \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\" (UID: \"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7\") "
Sep 11 04:09:37.902938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.902901    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb39139-5436-495b-bb0c-7014f4a7ea4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cb39139-5436-495b-bb0c-7014f4a7ea4c" (UID: "8cb39139-5436-495b-bb0c-7014f4a7ea4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:37.903060 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.902915    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7" (UID: "e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:09:37.904752 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.904723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb39139-5436-495b-bb0c-7014f4a7ea4c-kube-api-access-k5jdd" (OuterVolumeSpecName: "kube-api-access-k5jdd") pod "8cb39139-5436-495b-bb0c-7014f4a7ea4c" (UID: "8cb39139-5436-495b-bb0c-7014f4a7ea4c"). InnerVolumeSpecName "kube-api-access-k5jdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:37.904858 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:37.904757    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-kube-api-access-pr4x7" (OuterVolumeSpecName: "kube-api-access-pr4x7") pod "e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7" (UID: "e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7"). InnerVolumeSpecName "kube-api-access-pr4x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:09:38.003930 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.003895    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5jdd\" (UniqueName: \"kubernetes.io/projected/8cb39139-5436-495b-bb0c-7014f4a7ea4c-kube-api-access-k5jdd\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:09:38.003930 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.003921    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb39139-5436-495b-bb0c-7014f4a7ea4c-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:09:38.003930 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.003931    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr4x7\" (UniqueName: \"kubernetes.io/projected/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-kube-api-access-pr4x7\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:09:38.003930 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.003939    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:09:38.292938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.292905    2566 generic.go:356] "Generic (PLEG): container finished" podID="8cb39139-5436-495b-bb0c-7014f4a7ea4c" containerID="f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b" exitCode=0
Sep 11 04:09:38.293132 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.292954    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"
Sep 11 04:09:38.293132 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.292960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx" event={"ID":"8cb39139-5436-495b-bb0c-7014f4a7ea4c","Type":"ContainerDied","Data":"f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b"}
Sep 11 04:09:38.293132 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.293019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx" event={"ID":"8cb39139-5436-495b-bb0c-7014f4a7ea4c","Type":"ContainerDied","Data":"b1d58ecb2702320582af276ffb6709acf5a3699470e3df1a77d8d48f58e7d9b2"}
Sep 11 04:09:38.293132 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.293051    2566 scope.go:117] "RemoveContainer" containerID="f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b"
Sep 11 04:09:38.294512 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.294490    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7" containerID="2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b" exitCode=0
Sep 11 04:09:38.294624 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.294538    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"
Sep 11 04:09:38.294624 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.294579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw" event={"ID":"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7","Type":"ContainerDied","Data":"2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b"}
Sep 11 04:09:38.294624 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.294620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw" event={"ID":"e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7","Type":"ContainerDied","Data":"f453e3b85e17108c75f34aebb3f17ac7b3e3a3d2877e8983a0cdb2287394d5ca"}
Sep 11 04:09:38.309924 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.309906    2566 scope.go:117] "RemoveContainer" containerID="f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b"
Sep 11 04:09:38.310212 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:09:38.310193    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b\": container with ID starting with f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b not found: ID does not exist" containerID="f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b"
Sep 11 04:09:38.310262 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.310218    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b"} err="failed to get container status \"f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b\": rpc error: code = NotFound desc = could not find container \"f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b\": container with ID starting with f6b7e4135ff87129c860f6d57bc749f2f2b37aa5078dd9e89e36e881b0a4170b not found: ID does not exist"
Sep 11 04:09:38.310262 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.310238    2566 scope.go:117] "RemoveContainer" containerID="2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b"
Sep 11 04:09:38.317645 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.317628    2566 scope.go:117] "RemoveContainer" containerID="2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b"
Sep 11 04:09:38.317883 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:09:38.317865    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b\": container with ID starting with 2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b not found: ID does not exist" containerID="2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b"
Sep 11 04:09:38.317938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.317889    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b"} err="failed to get container status \"2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b\": rpc error: code = NotFound desc = could not find container \"2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b\": container with ID starting with 2ae0ce1e89e03375e31a1993f2102864ad19775a1eec681d67e62b2837f0f82b not found: ID does not exist"
Sep 11 04:09:38.387816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.387791    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"]
Sep 11 04:09:38.397286 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.397258    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0347a9fa-568c7b6445-lcdmx"]
Sep 11 04:09:38.427465 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.427446    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"]
Sep 11 04:09:38.433542 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.433521    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0347a9fa-c666ffb5d-zr5cw"]
Sep 11 04:09:38.613441 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.613364    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb39139-5436-495b-bb0c-7014f4a7ea4c" path="/var/lib/kubelet/pods/8cb39139-5436-495b-bb0c-7014f4a7ea4c/volumes"
Sep 11 04:09:38.613787 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:38.613772    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7" path="/var/lib/kubelet/pods/e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7/volumes"
Sep 11 04:09:46.061722 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:46.061686    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"]
Sep 11 04:09:54.378582 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:09:54.377770    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:10:01.080883 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.080835    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd" podUID="96bde8f6-b372-4df2-9336-cc7f61e7ddd0" containerName="main" containerID="cri-o://1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348" gracePeriod=45
Sep 11 04:10:01.333895 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.333838    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:10:01.404623 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.404599    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-dshm\") pod \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") "
Sep 11 04:10:01.404766 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.404634    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-model-cache\") pod \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") "
Sep 11 04:10:01.404766 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.404701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-home\") pod \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") "
Sep 11 04:10:01.404766 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.404754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tmp-dir\") pod \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") "
Sep 11 04:10:01.404903 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.404803    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrt8f\" (UniqueName: \"kubernetes.io/projected/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-kube-api-access-vrt8f\") pod \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") "
Sep 11 04:10:01.404903 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.404834    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tls-certs\") pod \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\" (UID: \"96bde8f6-b372-4df2-9336-cc7f61e7ddd0\") "
Sep 11 04:10:01.405022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.404921    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-home" (OuterVolumeSpecName: "home") pod "96bde8f6-b372-4df2-9336-cc7f61e7ddd0" (UID: "96bde8f6-b372-4df2-9336-cc7f61e7ddd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:01.405022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.404946    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "96bde8f6-b372-4df2-9336-cc7f61e7ddd0" (UID: "96bde8f6-b372-4df2-9336-cc7f61e7ddd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:01.405022 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.405010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96bde8f6-b372-4df2-9336-cc7f61e7ddd0" (UID: "96bde8f6-b372-4df2-9336-cc7f61e7ddd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:01.405246 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.405221    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-model-cache\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:01.405318 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.405247    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-home\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:01.405318 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.405261    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tmp-dir\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:01.406881 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.406854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-dshm" (OuterVolumeSpecName: "dshm") pod "96bde8f6-b372-4df2-9336-cc7f61e7ddd0" (UID: "96bde8f6-b372-4df2-9336-cc7f61e7ddd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:01.407000 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.406933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96bde8f6-b372-4df2-9336-cc7f61e7ddd0" (UID: "96bde8f6-b372-4df2-9336-cc7f61e7ddd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:10:01.407000 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.406948    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-kube-api-access-vrt8f" (OuterVolumeSpecName: "kube-api-access-vrt8f") pod "96bde8f6-b372-4df2-9336-cc7f61e7ddd0" (UID: "96bde8f6-b372-4df2-9336-cc7f61e7ddd0"). InnerVolumeSpecName "kube-api-access-vrt8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:10:01.416943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.416920    2566 generic.go:356] "Generic (PLEG): container finished" podID="96bde8f6-b372-4df2-9336-cc7f61e7ddd0" containerID="1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348" exitCode=0
Sep 11 04:10:01.417058 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.416957    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"
Sep 11 04:10:01.417058 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.417006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd" event={"ID":"96bde8f6-b372-4df2-9336-cc7f61e7ddd0","Type":"ContainerDied","Data":"1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348"}
Sep 11 04:10:01.417058 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.417054    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd" event={"ID":"96bde8f6-b372-4df2-9336-cc7f61e7ddd0","Type":"ContainerDied","Data":"c70725a936151e6f57ef80469cdbba1cb50e2aaae820ca8cd585452b95c5db55"}
Sep 11 04:10:01.417190 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.417078    2566 scope.go:117] "RemoveContainer" containerID="1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348"
Sep 11 04:10:01.426578 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.426562    2566 scope.go:117] "RemoveContainer" containerID="1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348"
Sep 11 04:10:01.426834 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:10:01.426816    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348\": container with ID starting with 1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348 not found: ID does not exist" containerID="1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348"
Sep 11 04:10:01.426897 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.426840    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348"} err="failed to get container status \"1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348\": rpc error: code = NotFound desc = could not find container \"1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348\": container with ID starting with 1163d3f6a2a0ebcdcc910ec5f00b6e7947f31506c4905895396bbe4661742348 not found: ID does not exist"
Sep 11 04:10:01.438669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.438647    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"]
Sep 11 04:10:01.449174 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.449146    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0347a9fa/e2e-lifecycle-model-0347a9fa-kserve-6bc5fbdb79-vwzmd"]
Sep 11 04:10:01.505730 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.505706    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrt8f\" (UniqueName: \"kubernetes.io/projected/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-kube-api-access-vrt8f\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:01.505730 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.505728    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-tls-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:01.505856 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:01.505738    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bde8f6-b372-4df2-9336-cc7f61e7ddd0-dshm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:02.612417 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:02.612371    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96bde8f6-b372-4df2-9336-cc7f61e7ddd0" path="/var/lib/kubelet/pods/96bde8f6-b372-4df2-9336-cc7f61e7ddd0/volumes"
Sep 11 04:10:02.652180 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:02.652157    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:10:02.652632 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:02.652614    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:10:09.982997 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:09.982955    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:10:19.621689 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.621640    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7489-0a756e"]
Sep 11 04:10:19.622294 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622273    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7" containerName="payload-pre-processing"
Sep 11 04:10:19.622386 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622297    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7" containerName="payload-pre-processing"
Sep 11 04:10:19.622386 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622314    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb39139-5436-495b-bb0c-7014f4a7ea4c" containerName="payload-processing"
Sep 11 04:10:19.622386 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622322    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb39139-5436-495b-bb0c-7014f4a7ea4c" containerName="payload-processing"
Sep 11 04:10:19.622386 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622359    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96bde8f6-b372-4df2-9336-cc7f61e7ddd0" containerName="main"
Sep 11 04:10:19.622386 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622368    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="96bde8f6-b372-4df2-9336-cc7f61e7ddd0" containerName="main"
Sep 11 04:10:19.622667 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622494    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="96bde8f6-b372-4df2-9336-cc7f61e7ddd0" containerName="main"
Sep 11 04:10:19.622667 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622508    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ba8e79-fa6e-420f-b561-e1d6a9d93ef7" containerName="payload-pre-processing"
Sep 11 04:10:19.622667 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.622520    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb39139-5436-495b-bb0c-7014f4a7ea4c" containerName="payload-processing"
Sep 11 04:10:19.627269 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.627247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:19.631144 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.631124    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mhkpw\""
Sep 11 04:10:19.642240 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.642207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7489-0a756e"]
Sep 11 04:10:19.663534 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.663503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nftfv\" (UniqueName: \"kubernetes.io/projected/ec886a5b-f0b6-45a8-a22f-98d50d6c6244-kube-api-access-nftfv\") pod \"test-curl-7489-0a756e\" (UID: \"ec886a5b-f0b6-45a8-a22f-98d50d6c6244\") " pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:19.764384 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.764359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nftfv\" (UniqueName: \"kubernetes.io/projected/ec886a5b-f0b6-45a8-a22f-98d50d6c6244-kube-api-access-nftfv\") pod \"test-curl-7489-0a756e\" (UID: \"ec886a5b-f0b6-45a8-a22f-98d50d6c6244\") " pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:19.776510 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.776488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nftfv\" (UniqueName: \"kubernetes.io/projected/ec886a5b-f0b6-45a8-a22f-98d50d6c6244-kube-api-access-nftfv\") pod \"test-curl-7489-0a756e\" (UID: \"ec886a5b-f0b6-45a8-a22f-98d50d6c6244\") " pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:19.936738 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:19.936647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:20.207433 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:20.207396    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7489-0a756e"]
Sep 11 04:10:20.209395 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:10:20.209368    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec886a5b_f0b6_45a8_a22f_98d50d6c6244.slice/crio-f16bcfbd517cf635261a841c5c4174976ee1c0f856a9442767c40bfe4620083b WatchSource:0}: Error finding container f16bcfbd517cf635261a841c5c4174976ee1c0f856a9442767c40bfe4620083b: Status 404 returned error can't find the container with id f16bcfbd517cf635261a841c5c4174976ee1c0f856a9442767c40bfe4620083b
Sep 11 04:10:20.510872 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:20.510843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7489-0a756e" event={"ID":"ec886a5b-f0b6-45a8-a22f-98d50d6c6244","Type":"ContainerStarted","Data":"36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed"}
Sep 11 04:10:20.510989 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:20.510883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7489-0a756e" event={"ID":"ec886a5b-f0b6-45a8-a22f-98d50d6c6244","Type":"ContainerStarted","Data":"f16bcfbd517cf635261a841c5c4174976ee1c0f856a9442767c40bfe4620083b"}
Sep 11 04:10:20.528119 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:20.528073    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7489-0a756e" podStartSLOduration=1.5280573899999998 podStartE2EDuration="1.52805739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:10:20.526119333 +0000 UTC m=+1218.392139075" watchObservedRunningTime="2026-09-11 04:10:20.52805739 +0000 UTC m=+1218.394077125"
Sep 11 04:10:21.141610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:21.141578    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7489-0a756e"]
Sep 11 04:10:21.146340 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:21.146316    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7489-0a756e"]
Sep 11 04:10:21.515935 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:10:21.515910    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7489-0a756e" container="test-curl-7489-0a756e"
Sep 11 04:10:22.520868 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:22.520834    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7489-0a756e" podUID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" containerName="test-curl-7489-0a756e" containerID="cri-o://36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed" gracePeriod=2
Sep 11 04:10:22.523365 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:10:22.523337    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7489-0a756e\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:22.611474 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:10:22.611443    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7489-0a756e\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:24.342829 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:24.342794    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"]
Sep 11 04:10:24.665004 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:24.664982    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:24.709107 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:24.709080    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nftfv\" (UniqueName: \"kubernetes.io/projected/ec886a5b-f0b6-45a8-a22f-98d50d6c6244-kube-api-access-nftfv\") pod \"ec886a5b-f0b6-45a8-a22f-98d50d6c6244\" (UID: \"ec886a5b-f0b6-45a8-a22f-98d50d6c6244\") "
Sep 11 04:10:24.710961 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:24.710934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec886a5b-f0b6-45a8-a22f-98d50d6c6244-kube-api-access-nftfv" (OuterVolumeSpecName: "kube-api-access-nftfv") pod "ec886a5b-f0b6-45a8-a22f-98d50d6c6244" (UID: "ec886a5b-f0b6-45a8-a22f-98d50d6c6244"). InnerVolumeSpecName "kube-api-access-nftfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:10:24.810017 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:24.809990    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nftfv\" (UniqueName: \"kubernetes.io/projected/ec886a5b-f0b6-45a8-a22f-98d50d6c6244-kube-api-access-nftfv\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:25.536367 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:25.536335    2566 generic.go:356] "Generic (PLEG): container finished" podID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" containerID="36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed" exitCode=137
Sep 11 04:10:25.536768 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:25.536372    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:25.536768 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:25.536450    2566 scope.go:117] "RemoveContainer" containerID="36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed"
Sep 11 04:10:25.545497 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:25.545478    2566 scope.go:117] "RemoveContainer" containerID="36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed"
Sep 11 04:10:25.545785 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:10:25.545763    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed\": container with ID starting with 36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed not found: ID does not exist" containerID="36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed"
Sep 11 04:10:25.545839 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:25.545800    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed"} err="failed to get container status \"36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed\": rpc error: code = NotFound desc = could not find container \"36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed\": container with ID starting with 36a63bb1b4b1c36e1e741ee4e7334edd4e8308ca68b4842b28234160d9e222ed not found: ID does not exist"
Sep 11 04:10:25.546903 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:10:25.546881    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7489-0a756e\" is forbidden: User \"system:node:ip-10-0-131-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-122.ec2.internal' and this object" podUID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" pod="opendatahub/test-curl-7489-0a756e"
Sep 11 04:10:25.579956 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:25.579933    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:10:26.611760 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:26.611729    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" path="/var/lib/kubelet/pods/ec886a5b-f0b6-45a8-a22f-98d50d6c6244/volumes"
Sep 11 04:10:36.917555 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.917525    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"]
Sep 11 04:10:36.918018 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.917900    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" containerName="test-curl-7489-0a756e"
Sep 11 04:10:36.918018 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.917911    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" containerName="test-curl-7489-0a756e"
Sep 11 04:10:36.918018 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.917970    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec886a5b-f0b6-45a8-a22f-98d50d6c6244" containerName="test-curl-7489-0a756e"
Sep 11 04:10:36.920969 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.920944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:36.925226 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.925202    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"openshift-service-ca.crt\""
Sep 11 04:10:36.925344 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.925322    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"default-dockercfg-x45hg\""
Sep 11 04:10:36.925344 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.925325    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"sub-test-model-9d031d-kserve-self-signed-certs\""
Sep 11 04:10:36.925482 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.925328    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9d031d\"/\"kube-root-ca.crt\""
Sep 11 04:10:36.930695 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:36.930673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"]
Sep 11 04:10:37.009401 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.009373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tmp-dir\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.009540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.009442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-dshm\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.009540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.009463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-model-cache\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.009540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.009482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfrn4\" (UniqueName: \"kubernetes.io/projected/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-kube-api-access-tfrn4\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.009655 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.009560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-home\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.009655 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.009578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tls-certs\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110109 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-home\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110292 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tls-certs\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110292 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tmp-dir\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110292 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-dshm\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110954 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-model-cache\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110954 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110553    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-home\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110954 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfrn4\" (UniqueName: \"kubernetes.io/projected/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-kube-api-access-tfrn4\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110954 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tmp-dir\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.110954 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.110800    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-model-cache\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.116306 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.116253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-dshm\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.116913 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.116889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tls-certs\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.125859 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.125837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfrn4\" (UniqueName: \"kubernetes.io/projected/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-kube-api-access-tfrn4\") pod \"sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") " pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.231192 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.231168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:37.501846 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.501821    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"]
Sep 11 04:10:37.503445 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:10:37.503387    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod348bf11c_6015_4afb_ae3c_aec6c3ce02e1.slice/crio-1f9d8970590955d596eb1b7d1aaf3c2af3e9e1ba85ae816704e14eda43606f69 WatchSource:0}: Error finding container 1f9d8970590955d596eb1b7d1aaf3c2af3e9e1ba85ae816704e14eda43606f69: Status 404 returned error can't find the container with id 1f9d8970590955d596eb1b7d1aaf3c2af3e9e1ba85ae816704e14eda43606f69
Sep 11 04:10:37.598108 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.598081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct" event={"ID":"348bf11c-6015-4afb-ae3c-aec6c3ce02e1","Type":"ContainerStarted","Data":"a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179"}
Sep 11 04:10:37.598246 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.598117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct" event={"ID":"348bf11c-6015-4afb-ae3c-aec6c3ce02e1","Type":"ContainerStarted","Data":"1f9d8970590955d596eb1b7d1aaf3c2af3e9e1ba85ae816704e14eda43606f69"}
Sep 11 04:10:37.615438 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:37.615378    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct" podStartSLOduration=1.615363186 podStartE2EDuration="1.615363186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:10:37.613881103 +0000 UTC m=+1235.479900842" watchObservedRunningTime="2026-09-11 04:10:37.615363186 +0000 UTC m=+1235.481382923"
Sep 11 04:10:39.361526 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.361483    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv" podUID="73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" containerName="main" containerID="cri-o://f09719ad226a35ce01d4c6c53e549aa58521c9bf04026423aee01cd69bb7689d" gracePeriod=45
Sep 11 04:10:39.611324 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.611299    2566 generic.go:356] "Generic (PLEG): container finished" podID="73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" containerID="f09719ad226a35ce01d4c6c53e549aa58521c9bf04026423aee01cd69bb7689d" exitCode=0
Sep 11 04:10:39.611470 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.611369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv" event={"ID":"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052","Type":"ContainerDied","Data":"f09719ad226a35ce01d4c6c53e549aa58521c9bf04026423aee01cd69bb7689d"}
Sep 11 04:10:39.611470 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.611429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv" event={"ID":"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052","Type":"ContainerDied","Data":"25a07f165d501ddb4f4c4769963345a0f9992d6cca6a7e2ec52a425a849e310e"}
Sep 11 04:10:39.611470 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.611445    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a07f165d501ddb4f4c4769963345a0f9992d6cca6a7e2ec52a425a849e310e"
Sep 11 04:10:39.617242 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.617194    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:10:39.733282 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733256    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-dshm\") pod \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") "
Sep 11 04:10:39.733427 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733308    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tls-certs\") pod \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") "
Sep 11 04:10:39.733427 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733333    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb9vs\" (UniqueName: \"kubernetes.io/projected/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-kube-api-access-sb9vs\") pod \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") "
Sep 11 04:10:39.733427 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733353    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tmp-dir\") pod \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") "
Sep 11 04:10:39.733427 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733393    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-model-cache\") pod \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") "
Sep 11 04:10:39.733666 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-home\") pod \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\" (UID: \"73ab450d-e9f2-4cfb-8dc8-0e964e6a2052\") "
Sep 11 04:10:39.733807 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-model-cache" (OuterVolumeSpecName: "model-cache") pod "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" (UID: "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:39.733994 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733799    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" (UID: "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:39.733994 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.733863    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-home" (OuterVolumeSpecName: "home") pod "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" (UID: "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:39.735648 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.735625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-kube-api-access-sb9vs" (OuterVolumeSpecName: "kube-api-access-sb9vs") pod "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" (UID: "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052"). InnerVolumeSpecName "kube-api-access-sb9vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:10:39.735842 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.735822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-dshm" (OuterVolumeSpecName: "dshm") pod "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" (UID: "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:10:39.735842 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.735817    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" (UID: "73ab450d-e9f2-4cfb-8dc8-0e964e6a2052"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:10:39.834525 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.834499    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tls-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:39.834525 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.834523    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb9vs\" (UniqueName: \"kubernetes.io/projected/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-kube-api-access-sb9vs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:39.834661 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.834537    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-tmp-dir\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:39.834661 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.834551    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-model-cache\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:39.834661 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.834562    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-home\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:39.834661 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:39.834573    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052-dshm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:10:40.616976 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:40.616939    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"
Sep 11 04:10:40.639288 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:40.639252    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"]
Sep 11 04:10:40.643105 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:40.643076    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c158b2/auth-test-model-c158b2-kserve-76559d864c-sjprv"]
Sep 11 04:10:42.613553 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:42.613520    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" path="/var/lib/kubelet/pods/73ab450d-e9f2-4cfb-8dc8-0e964e6a2052/volumes"
Sep 11 04:10:43.084338 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:43.084302    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:10:47.232030 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:47.231999    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:47.232437 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:47.232042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:47.244099 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:47.244079    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:10:47.666200 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:10:47.666122    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:11:06.980190 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:06.980158    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:11:22.683303 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:22.683272    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:11:28.718859 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.718825    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"]
Sep 11 04:11:28.719216 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.719206    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" containerName="main"
Sep 11 04:11:28.719216 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.719217    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" containerName="main"
Sep 11 04:11:28.719292 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.719287    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73ab450d-e9f2-4cfb-8dc8-0e964e6a2052" containerName="main"
Sep 11 04:11:28.723596 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.723579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:28.725987 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.725970    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a7eaac\""
Sep 11 04:11:28.735370 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.735346    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"]
Sep 11 04:11:28.856239 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.856204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtbp\" (UniqueName: \"kubernetes.io/projected/f216b913-95a5-4d2d-a187-afc430bcaae4-kube-api-access-fmtbp\") pod \"payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2\" (UID: \"f216b913-95a5-4d2d-a187-afc430bcaae4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:28.856659 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.856632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f216b913-95a5-4d2d-a187-afc430bcaae4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2\" (UID: \"f216b913-95a5-4d2d-a187-afc430bcaae4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:28.957311 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.957281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtbp\" (UniqueName: \"kubernetes.io/projected/f216b913-95a5-4d2d-a187-afc430bcaae4-kube-api-access-fmtbp\") pod \"payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2\" (UID: \"f216b913-95a5-4d2d-a187-afc430bcaae4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:28.957485 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.957441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f216b913-95a5-4d2d-a187-afc430bcaae4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2\" (UID: \"f216b913-95a5-4d2d-a187-afc430bcaae4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:28.958066 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.958047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f216b913-95a5-4d2d-a187-afc430bcaae4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2\" (UID: \"f216b913-95a5-4d2d-a187-afc430bcaae4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:28.965820 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:28.965791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtbp\" (UniqueName: \"kubernetes.io/projected/f216b913-95a5-4d2d-a187-afc430bcaae4-kube-api-access-fmtbp\") pod \"payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2\" (UID: \"f216b913-95a5-4d2d-a187-afc430bcaae4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:29.034016 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:29.033959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:29.330464 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:29.330435    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"]
Sep 11 04:11:29.332711 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:11:29.332684    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf216b913_95a5_4d2d_a187_afc430bcaae4.slice/crio-fd68ef1736147a77904329612b773c661c1ef03be4e77b0f711f588b5bfbfddc WatchSource:0}: Error finding container fd68ef1736147a77904329612b773c661c1ef03be4e77b0f711f588b5bfbfddc: Status 404 returned error can't find the container with id fd68ef1736147a77904329612b773c661c1ef03be4e77b0f711f588b5bfbfddc
Sep 11 04:11:29.866158 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:29.866123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2" event={"ID":"f216b913-95a5-4d2d-a187-afc430bcaae4","Type":"ContainerStarted","Data":"b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043"}
Sep 11 04:11:29.866576 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:29.866173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2" event={"ID":"f216b913-95a5-4d2d-a187-afc430bcaae4","Type":"ContainerStarted","Data":"fd68ef1736147a77904329612b773c661c1ef03be4e77b0f711f588b5bfbfddc"}
Sep 11 04:11:29.866576 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:29.866233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:29.882949 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:29.882904    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2" podStartSLOduration=1.882888789 podStartE2EDuration="1.882888789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:29.881555806 +0000 UTC m=+1287.747575543" watchObservedRunningTime="2026-09-11 04:11:29.882888789 +0000 UTC m=+1287.748908525"
Sep 11 04:11:38.585236 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:38.585201    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:11:40.873039 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:40.873003    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:11:42.377373 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.377341    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7489-6026e2"]
Sep 11 04:11:42.380895 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.380877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-6026e2"
Sep 11 04:11:42.383861 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.383840    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mhkpw\""
Sep 11 04:11:42.389820 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.389792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7489-6026e2"]
Sep 11 04:11:42.480312 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.480281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29sp6\" (UniqueName: \"kubernetes.io/projected/3865ea34-0b1d-43b7-a383-8321c13b4bb6-kube-api-access-29sp6\") pod \"test-curl-7489-6026e2\" (UID: \"3865ea34-0b1d-43b7-a383-8321c13b4bb6\") " pod="opendatahub/test-curl-7489-6026e2"
Sep 11 04:11:42.581401 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.581370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29sp6\" (UniqueName: \"kubernetes.io/projected/3865ea34-0b1d-43b7-a383-8321c13b4bb6-kube-api-access-29sp6\") pod \"test-curl-7489-6026e2\" (UID: \"3865ea34-0b1d-43b7-a383-8321c13b4bb6\") " pod="opendatahub/test-curl-7489-6026e2"
Sep 11 04:11:42.590397 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.590372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29sp6\" (UniqueName: \"kubernetes.io/projected/3865ea34-0b1d-43b7-a383-8321c13b4bb6-kube-api-access-29sp6\") pod \"test-curl-7489-6026e2\" (UID: \"3865ea34-0b1d-43b7-a383-8321c13b4bb6\") " pod="opendatahub/test-curl-7489-6026e2"
Sep 11 04:11:42.691494 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.691424    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-6026e2"
Sep 11 04:11:42.963706 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:42.963678    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7489-6026e2"]
Sep 11 04:11:42.967023 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:11:42.966996    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3865ea34_0b1d_43b7_a383_8321c13b4bb6.slice/crio-26316cb836cd09cf38980bc51f7a726f5d96a6e3c106d47033f1bc421d39d188 WatchSource:0}: Error finding container 26316cb836cd09cf38980bc51f7a726f5d96a6e3c106d47033f1bc421d39d188: Status 404 returned error can't find the container with id 26316cb836cd09cf38980bc51f7a726f5d96a6e3c106d47033f1bc421d39d188
Sep 11 04:11:43.940289 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:43.940255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7489-6026e2" event={"ID":"3865ea34-0b1d-43b7-a383-8321c13b4bb6","Type":"ContainerStarted","Data":"7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d"}
Sep 11 04:11:43.940781 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:43.940298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7489-6026e2" event={"ID":"3865ea34-0b1d-43b7-a383-8321c13b4bb6","Type":"ContainerStarted","Data":"26316cb836cd09cf38980bc51f7a726f5d96a6e3c106d47033f1bc421d39d188"}
Sep 11 04:11:43.955747 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:43.955698    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7489-6026e2" podStartSLOduration=1.9556833999999998 podStartE2EDuration="1.9556834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:43.953651266 +0000 UTC m=+1301.819671002" watchObservedRunningTime="2026-09-11 04:11:43.9556834 +0000 UTC m=+1301.821703136"
Sep 11 04:11:44.537642 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.537607    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7489-6026e2"]
Sep 11 04:11:44.542029 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.542002    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7489-6026e2"]
Sep 11 04:11:44.658882 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.658809    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7489-1ed898"]
Sep 11 04:11:44.659193 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.659179    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3865ea34-0b1d-43b7-a383-8321c13b4bb6" containerName="test-curl-7489-6026e2"
Sep 11 04:11:44.659238 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.659195    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3865ea34-0b1d-43b7-a383-8321c13b4bb6" containerName="test-curl-7489-6026e2"
Sep 11 04:11:44.659272 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.659262    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3865ea34-0b1d-43b7-a383-8321c13b4bb6" containerName="test-curl-7489-6026e2"
Sep 11 04:11:44.661914 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.661899    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-1ed898"
Sep 11 04:11:44.669187 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.669166    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7489-1ed898"]
Sep 11 04:11:44.804590 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.804555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbtj\" (UniqueName: \"kubernetes.io/projected/c3a876f9-bb05-44b2-ab84-3da1d81bfb22-kube-api-access-hwbtj\") pod \"test-curl-7489-1ed898\" (UID: \"c3a876f9-bb05-44b2-ab84-3da1d81bfb22\") " pod="opendatahub/test-curl-7489-1ed898"
Sep 11 04:11:44.905193 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.905152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbtj\" (UniqueName: \"kubernetes.io/projected/c3a876f9-bb05-44b2-ab84-3da1d81bfb22-kube-api-access-hwbtj\") pod \"test-curl-7489-1ed898\" (UID: \"c3a876f9-bb05-44b2-ab84-3da1d81bfb22\") " pod="opendatahub/test-curl-7489-1ed898"
Sep 11 04:11:44.914467 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.914396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbtj\" (UniqueName: \"kubernetes.io/projected/c3a876f9-bb05-44b2-ab84-3da1d81bfb22-kube-api-access-hwbtj\") pod \"test-curl-7489-1ed898\" (UID: \"c3a876f9-bb05-44b2-ab84-3da1d81bfb22\") " pod="opendatahub/test-curl-7489-1ed898"
Sep 11 04:11:44.945809 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:11:44.945788    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7489-6026e2" container="test-curl-7489-6026e2"
Sep 11 04:11:44.971915 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:44.971897    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-1ed898"
Sep 11 04:11:45.240601 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:45.240576    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7489-1ed898"]
Sep 11 04:11:45.242145 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:11:45.242115    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3a876f9_bb05_44b2_ab84_3da1d81bfb22.slice/crio-5791781a829413a016a1c37f21852675666e53baf846f14c79fb3652a7562796 WatchSource:0}: Error finding container 5791781a829413a016a1c37f21852675666e53baf846f14c79fb3652a7562796: Status 404 returned error can't find the container with id 5791781a829413a016a1c37f21852675666e53baf846f14c79fb3652a7562796
Sep 11 04:11:45.953088 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:45.953051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7489-1ed898" event={"ID":"c3a876f9-bb05-44b2-ab84-3da1d81bfb22","Type":"ContainerStarted","Data":"ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127"}
Sep 11 04:11:45.953088 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:45.953095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7489-1ed898" event={"ID":"c3a876f9-bb05-44b2-ab84-3da1d81bfb22","Type":"ContainerStarted","Data":"5791781a829413a016a1c37f21852675666e53baf846f14c79fb3652a7562796"}
Sep 11 04:11:45.953597 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:45.953159    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7489-6026e2" podUID="3865ea34-0b1d-43b7-a383-8321c13b4bb6" containerName="test-curl-7489-6026e2" containerID="cri-o://7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d" gracePeriod=2
Sep 11 04:11:45.968323 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:45.968275    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7489-1ed898" podStartSLOduration=1.968258754 podStartE2EDuration="1.968258754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:45.96707227 +0000 UTC m=+1303.833092005" watchObservedRunningTime="2026-09-11 04:11:45.968258754 +0000 UTC m=+1303.834278492"
Sep 11 04:11:46.521503 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:46.521473    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7489-1ed898"]
Sep 11 04:11:46.525308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:46.525272    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7489-1ed898"]
Sep 11 04:11:46.958925 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:11:46.958894    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7489-1ed898" container="test-curl-7489-1ed898"
Sep 11 04:11:47.963540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:47.963507    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7489-1ed898" podUID="c3a876f9-bb05-44b2-ab84-3da1d81bfb22" containerName="test-curl-7489-1ed898" containerID="cri-o://ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127" gracePeriod=2
Sep 11 04:11:48.144257 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.144233    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-6026e2"
Sep 11 04:11:48.232659 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.232635    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29sp6\" (UniqueName: \"kubernetes.io/projected/3865ea34-0b1d-43b7-a383-8321c13b4bb6-kube-api-access-29sp6\") pod \"3865ea34-0b1d-43b7-a383-8321c13b4bb6\" (UID: \"3865ea34-0b1d-43b7-a383-8321c13b4bb6\") "
Sep 11 04:11:48.234579 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.234557    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3865ea34-0b1d-43b7-a383-8321c13b4bb6-kube-api-access-29sp6" (OuterVolumeSpecName: "kube-api-access-29sp6") pod "3865ea34-0b1d-43b7-a383-8321c13b4bb6" (UID: "3865ea34-0b1d-43b7-a383-8321c13b4bb6"). InnerVolumeSpecName "kube-api-access-29sp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:11:48.333248 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.333226    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29sp6\" (UniqueName: \"kubernetes.io/projected/3865ea34-0b1d-43b7-a383-8321c13b4bb6-kube-api-access-29sp6\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:11:48.478650 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.478620    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"]
Sep 11 04:11:48.621257 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.621180    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3865ea34-0b1d-43b7-a383-8321c13b4bb6" path="/var/lib/kubelet/pods/3865ea34-0b1d-43b7-a383-8321c13b4bb6/volumes"
Sep 11 04:11:48.969786 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.969740    2566 generic.go:356] "Generic (PLEG): container finished" podID="3865ea34-0b1d-43b7-a383-8321c13b4bb6" containerID="7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d" exitCode=137
Sep 11 04:11:48.970146 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.969797    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-6026e2"
Sep 11 04:11:48.970146 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.969846    2566 scope.go:117] "RemoveContainer" containerID="7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d"
Sep 11 04:11:48.978144 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.978125    2566 scope.go:117] "RemoveContainer" containerID="7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d"
Sep 11 04:11:48.978442 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:11:48.978395    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d\": container with ID starting with 7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d not found: ID does not exist" containerID="7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d"
Sep 11 04:11:48.978538 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:48.978456    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d"} err="failed to get container status \"7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d\": rpc error: code = NotFound desc = could not find container \"7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d\": container with ID starting with 7bc3da80daef4772a8ba8991c9ea9ad3965c83d8f18ae3d7b5b8f7a5a217ff0d not found: ID does not exist"
Sep 11 04:11:50.148845 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.148822    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-1ed898"
Sep 11 04:11:50.250387 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.250321    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwbtj\" (UniqueName: \"kubernetes.io/projected/c3a876f9-bb05-44b2-ab84-3da1d81bfb22-kube-api-access-hwbtj\") pod \"c3a876f9-bb05-44b2-ab84-3da1d81bfb22\" (UID: \"c3a876f9-bb05-44b2-ab84-3da1d81bfb22\") "
Sep 11 04:11:50.252377 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.252352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a876f9-bb05-44b2-ab84-3da1d81bfb22-kube-api-access-hwbtj" (OuterVolumeSpecName: "kube-api-access-hwbtj") pod "c3a876f9-bb05-44b2-ab84-3da1d81bfb22" (UID: "c3a876f9-bb05-44b2-ab84-3da1d81bfb22"). InnerVolumeSpecName "kube-api-access-hwbtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:11:50.351841 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.351814    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbtj\" (UniqueName: \"kubernetes.io/projected/c3a876f9-bb05-44b2-ab84-3da1d81bfb22-kube-api-access-hwbtj\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:11:50.612727 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.612634    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a876f9-bb05-44b2-ab84-3da1d81bfb22" path="/var/lib/kubelet/pods/c3a876f9-bb05-44b2-ab84-3da1d81bfb22/volumes"
Sep 11 04:11:50.981998 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.981966    2566 generic.go:356] "Generic (PLEG): container finished" podID="c3a876f9-bb05-44b2-ab84-3da1d81bfb22" containerID="ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127" exitCode=137
Sep 11 04:11:50.982173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.982010    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7489-1ed898"
Sep 11 04:11:50.982173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.982062    2566 scope.go:117] "RemoveContainer" containerID="ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127"
Sep 11 04:11:50.990162 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.990145    2566 scope.go:117] "RemoveContainer" containerID="ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127"
Sep 11 04:11:50.990444 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:11:50.990424    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127\": container with ID starting with ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127 not found: ID does not exist" containerID="ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127"
Sep 11 04:11:50.990513 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:50.990450    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127"} err="failed to get container status \"ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127\": rpc error: code = NotFound desc = could not find container \"ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127\": container with ID starting with ae44cb0260d27d9a2d035b666c6943ea40617a1918e2b530a527ebe2ecc2f127 not found: ID does not exist"
Sep 11 04:11:52.787688 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:52.787654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:11:57.804605 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.804568    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"]
Sep 11 04:11:57.805005 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.804992    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a876f9-bb05-44b2-ab84-3da1d81bfb22" containerName="test-curl-7489-1ed898"
Sep 11 04:11:57.805056 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.805007    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a876f9-bb05-44b2-ab84-3da1d81bfb22" containerName="test-curl-7489-1ed898"
Sep 11 04:11:57.805091 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.805071    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a876f9-bb05-44b2-ab84-3da1d81bfb22" containerName="test-curl-7489-1ed898"
Sep 11 04:11:57.809513 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.809494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:57.811954 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.811909    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4c6f83\""
Sep 11 04:11:57.812090 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.811967    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4c6f83-dockercfg-6ftgv\""
Sep 11 04:11:57.820498 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.820479    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"]
Sep 11 04:11:57.822914 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.822893    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"]
Sep 11 04:11:57.826192 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.826172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:57.837299 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.837278    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"]
Sep 11 04:11:57.918560 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.918530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7822b86-770b-49d9-8554-e6b16c8d31f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz\" (UID: \"a7822b86-770b-49d9-8554-e6b16c8d31f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:57.918761 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.918597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b71b87-d7f2-4a70-a494-deba43f36d29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9\" (UID: \"37b71b87-d7f2-4a70-a494-deba43f36d29\") " pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:57.918761 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.918689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sl2x\" (UniqueName: \"kubernetes.io/projected/a7822b86-770b-49d9-8554-e6b16c8d31f7-kube-api-access-9sl2x\") pod \"payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz\" (UID: \"a7822b86-770b-49d9-8554-e6b16c8d31f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:57.918761 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:57.918728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qjd\" (UniqueName: \"kubernetes.io/projected/37b71b87-d7f2-4a70-a494-deba43f36d29-kube-api-access-k6qjd\") pod \"payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9\" (UID: \"37b71b87-d7f2-4a70-a494-deba43f36d29\") " pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:58.020155 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.020118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7822b86-770b-49d9-8554-e6b16c8d31f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz\" (UID: \"a7822b86-770b-49d9-8554-e6b16c8d31f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:58.020356 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.020188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b71b87-d7f2-4a70-a494-deba43f36d29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9\" (UID: \"37b71b87-d7f2-4a70-a494-deba43f36d29\") " pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:58.020356 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.020216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sl2x\" (UniqueName: \"kubernetes.io/projected/a7822b86-770b-49d9-8554-e6b16c8d31f7-kube-api-access-9sl2x\") pod \"payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz\" (UID: \"a7822b86-770b-49d9-8554-e6b16c8d31f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:58.020356 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.020249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qjd\" (UniqueName: \"kubernetes.io/projected/37b71b87-d7f2-4a70-a494-deba43f36d29-kube-api-access-k6qjd\") pod \"payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9\" (UID: \"37b71b87-d7f2-4a70-a494-deba43f36d29\") " pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:58.020899 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.020880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7822b86-770b-49d9-8554-e6b16c8d31f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz\" (UID: \"a7822b86-770b-49d9-8554-e6b16c8d31f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:58.020944 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.020900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b71b87-d7f2-4a70-a494-deba43f36d29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9\" (UID: \"37b71b87-d7f2-4a70-a494-deba43f36d29\") " pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:58.029202 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.029183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sl2x\" (UniqueName: \"kubernetes.io/projected/a7822b86-770b-49d9-8554-e6b16c8d31f7-kube-api-access-9sl2x\") pod \"payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz\" (UID: \"a7822b86-770b-49d9-8554-e6b16c8d31f7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:58.029283 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.029214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qjd\" (UniqueName: \"kubernetes.io/projected/37b71b87-d7f2-4a70-a494-deba43f36d29-kube-api-access-k6qjd\") pod \"payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9\" (UID: \"37b71b87-d7f2-4a70-a494-deba43f36d29\") " pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:58.121343 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.121271    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:58.136610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.136586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:58.400291 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.400262    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"]
Sep 11 04:11:58.401919 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:11:58.401894    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7822b86_770b_49d9_8554_e6b16c8d31f7.slice/crio-c19b2cabc1e1cf352585750dc6ef5e00ded083217e1084ac52c144f069c43de4 WatchSource:0}: Error finding container c19b2cabc1e1cf352585750dc6ef5e00ded083217e1084ac52c144f069c43de4: Status 404 returned error can't find the container with id c19b2cabc1e1cf352585750dc6ef5e00ded083217e1084ac52c144f069c43de4
Sep 11 04:11:58.418418 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:58.418378    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"]
Sep 11 04:11:58.419857 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:11:58.419834    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b71b87_d7f2_4a70_a494_deba43f36d29.slice/crio-eeb7d7c061aacbdf7f956d943578e9fe94c90bb7d3dafa9419454adfcc859f99 WatchSource:0}: Error finding container eeb7d7c061aacbdf7f956d943578e9fe94c90bb7d3dafa9419454adfcc859f99: Status 404 returned error can't find the container with id eeb7d7c061aacbdf7f956d943578e9fe94c90bb7d3dafa9419454adfcc859f99
Sep 11 04:11:59.022862 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:59.022828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz" event={"ID":"a7822b86-770b-49d9-8554-e6b16c8d31f7","Type":"ContainerStarted","Data":"7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a"}
Sep 11 04:11:59.023239 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:59.022892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:11:59.023239 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:59.022915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz" event={"ID":"a7822b86-770b-49d9-8554-e6b16c8d31f7","Type":"ContainerStarted","Data":"c19b2cabc1e1cf352585750dc6ef5e00ded083217e1084ac52c144f069c43de4"}
Sep 11 04:11:59.024540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:59.024510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9" event={"ID":"37b71b87-d7f2-4a70-a494-deba43f36d29","Type":"ContainerStarted","Data":"f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a"}
Sep 11 04:11:59.024675 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:59.024565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9" event={"ID":"37b71b87-d7f2-4a70-a494-deba43f36d29","Type":"ContainerStarted","Data":"eeb7d7c061aacbdf7f956d943578e9fe94c90bb7d3dafa9419454adfcc859f99"}
Sep 11 04:11:59.024675 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:59.024595    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:11:59.039191 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:59.039140    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz" podStartSLOduration=2.039121706 podStartE2EDuration="2.039121706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:59.038687744 +0000 UTC m=+1316.904707481" watchObservedRunningTime="2026-09-11 04:11:59.039121706 +0000 UTC m=+1316.905141442"
Sep 11 04:11:59.055085 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:11:59.055036    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9" podStartSLOduration=2.055023616 podStartE2EDuration="2.055023616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:59.053562717 +0000 UTC m=+1316.919582452" watchObservedRunningTime="2026-09-11 04:11:59.055023616 +0000 UTC m=+1316.921043384"
Sep 11 04:12:03.497584 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.497520    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct" podUID="348bf11c-6015-4afb-ae3c-aec6c3ce02e1" containerName="main" containerID="cri-o://a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179" gracePeriod=45
Sep 11 04:12:03.754288 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.754230    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:12:03.876907 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.876883    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-home\") pod \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") "
Sep 11 04:12:03.877051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.876921    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfrn4\" (UniqueName: \"kubernetes.io/projected/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-kube-api-access-tfrn4\") pod \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") "
Sep 11 04:12:03.877051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.876956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-model-cache\") pod \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") "
Sep 11 04:12:03.877051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.877013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tls-certs\") pod \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") "
Sep 11 04:12:03.877051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.877033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-dshm\") pod \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") "
Sep 11 04:12:03.877051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.877052    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tmp-dir\") pod \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\" (UID: \"348bf11c-6015-4afb-ae3c-aec6c3ce02e1\") "
Sep 11 04:12:03.877328 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.877189    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-home" (OuterVolumeSpecName: "home") pod "348bf11c-6015-4afb-ae3c-aec6c3ce02e1" (UID: "348bf11c-6015-4afb-ae3c-aec6c3ce02e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:03.877328 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.877221    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "348bf11c-6015-4afb-ae3c-aec6c3ce02e1" (UID: "348bf11c-6015-4afb-ae3c-aec6c3ce02e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:03.877452 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.877340    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-home\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:12:03.877452 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.877354    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-model-cache\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:12:03.877452 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.877382    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "348bf11c-6015-4afb-ae3c-aec6c3ce02e1" (UID: "348bf11c-6015-4afb-ae3c-aec6c3ce02e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:03.879254 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.879223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-kube-api-access-tfrn4" (OuterVolumeSpecName: "kube-api-access-tfrn4") pod "348bf11c-6015-4afb-ae3c-aec6c3ce02e1" (UID: "348bf11c-6015-4afb-ae3c-aec6c3ce02e1"). InnerVolumeSpecName "kube-api-access-tfrn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:03.879254 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.879225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-dshm" (OuterVolumeSpecName: "dshm") pod "348bf11c-6015-4afb-ae3c-aec6c3ce02e1" (UID: "348bf11c-6015-4afb-ae3c-aec6c3ce02e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:12:03.879254 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.879248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "348bf11c-6015-4afb-ae3c-aec6c3ce02e1" (UID: "348bf11c-6015-4afb-ae3c-aec6c3ce02e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:12:03.977868 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.977838    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfrn4\" (UniqueName: \"kubernetes.io/projected/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-kube-api-access-tfrn4\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:12:03.977868 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.977864    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tls-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:12:03.978008 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.977878    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-dshm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:12:03.978008 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:03.977890    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348bf11c-6015-4afb-ae3c-aec6c3ce02e1-tmp-dir\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:12:04.053145 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.053073    2566 generic.go:356] "Generic (PLEG): container finished" podID="348bf11c-6015-4afb-ae3c-aec6c3ce02e1" containerID="a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179" exitCode=0
Sep 11 04:12:04.053145 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.053118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct" event={"ID":"348bf11c-6015-4afb-ae3c-aec6c3ce02e1","Type":"ContainerDied","Data":"a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179"}
Sep 11 04:12:04.053145 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.053126    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"
Sep 11 04:12:04.053366 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.053149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct" event={"ID":"348bf11c-6015-4afb-ae3c-aec6c3ce02e1","Type":"ContainerDied","Data":"1f9d8970590955d596eb1b7d1aaf3c2af3e9e1ba85ae816704e14eda43606f69"}
Sep 11 04:12:04.053366 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.053167    2566 scope.go:117] "RemoveContainer" containerID="a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179"
Sep 11 04:12:04.063731 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.063716    2566 scope.go:117] "RemoveContainer" containerID="a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179"
Sep 11 04:12:04.063992 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:12:04.063974    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179\": container with ID starting with a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179 not found: ID does not exist" containerID="a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179"
Sep 11 04:12:04.064058 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.064000    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179"} err="failed to get container status \"a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179\": rpc error: code = NotFound desc = could not find container \"a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179\": container with ID starting with a9022ebf95a9bc965100fe3a9a47760c1bbd38e453f835cd4bcb415e17317179 not found: ID does not exist"
Sep 11 04:12:04.075086 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.075063    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"]
Sep 11 04:12:04.078489 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.078465    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9d031d/sub-test-model-9d031d-kserve-7d86b9cbc7-k95ct"]
Sep 11 04:12:04.612299 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:04.612268    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348bf11c-6015-4afb-ae3c-aec6c3ce02e1" path="/var/lib/kubelet/pods/348bf11c-6015-4afb-ae3c-aec6c3ce02e1/volumes"
Sep 11 04:12:10.033950 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:10.033922    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:12:10.034323 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:10.033986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:12:10.904422 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:10.904376    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:12:39.205648 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.205609    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"]
Sep 11 04:12:39.206533 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.206474    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2" podUID="f216b913-95a5-4d2d-a187-afc430bcaae4" containerName="payload-pre-processing" containerID="cri-o://b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043" gracePeriod=30
Sep 11 04:12:39.471881 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.471852    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:12:39.587579 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.587550    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmtbp\" (UniqueName: \"kubernetes.io/projected/f216b913-95a5-4d2d-a187-afc430bcaae4-kube-api-access-fmtbp\") pod \"f216b913-95a5-4d2d-a187-afc430bcaae4\" (UID: \"f216b913-95a5-4d2d-a187-afc430bcaae4\") "
Sep 11 04:12:39.587744 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.587667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f216b913-95a5-4d2d-a187-afc430bcaae4-plugins-config-volume\") pod \"f216b913-95a5-4d2d-a187-afc430bcaae4\" (UID: \"f216b913-95a5-4d2d-a187-afc430bcaae4\") "
Sep 11 04:12:39.588021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.587999    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f216b913-95a5-4d2d-a187-afc430bcaae4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f216b913-95a5-4d2d-a187-afc430bcaae4" (UID: "f216b913-95a5-4d2d-a187-afc430bcaae4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:12:39.589702 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.589679    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f216b913-95a5-4d2d-a187-afc430bcaae4-kube-api-access-fmtbp" (OuterVolumeSpecName: "kube-api-access-fmtbp") pod "f216b913-95a5-4d2d-a187-afc430bcaae4" (UID: "f216b913-95a5-4d2d-a187-afc430bcaae4"). InnerVolumeSpecName "kube-api-access-fmtbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:12:39.688663 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.688635    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f216b913-95a5-4d2d-a187-afc430bcaae4-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:12:39.688663 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:39.688658    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmtbp\" (UniqueName: \"kubernetes.io/projected/f216b913-95a5-4d2d-a187-afc430bcaae4-kube-api-access-fmtbp\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:12:40.237752 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.237712    2566 generic.go:356] "Generic (PLEG): container finished" podID="f216b913-95a5-4d2d-a187-afc430bcaae4" containerID="b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043" exitCode=0
Sep 11 04:12:40.238182 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.237759    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"
Sep 11 04:12:40.238182 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.237796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2" event={"ID":"f216b913-95a5-4d2d-a187-afc430bcaae4","Type":"ContainerDied","Data":"b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043"}
Sep 11 04:12:40.238182 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.237846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2" event={"ID":"f216b913-95a5-4d2d-a187-afc430bcaae4","Type":"ContainerDied","Data":"fd68ef1736147a77904329612b773c661c1ef03be4e77b0f711f588b5bfbfddc"}
Sep 11 04:12:40.238182 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.237868    2566 scope.go:117] "RemoveContainer" containerID="b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043"
Sep 11 04:12:40.247567 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.247545    2566 scope.go:117] "RemoveContainer" containerID="b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043"
Sep 11 04:12:40.247912 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:12:40.247893    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043\": container with ID starting with b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043 not found: ID does not exist" containerID="b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043"
Sep 11 04:12:40.247995 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.247921    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043"} err="failed to get container status \"b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043\": rpc error: code = NotFound desc = could not find container \"b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043\": container with ID starting with b95bc88994d8f6dcb757f0ffb054fa6ed2c9e5912d61c3a37d4ba20871d3e043 not found: ID does not exist"
Sep 11 04:12:40.260721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.260693    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"]
Sep 11 04:12:40.267341 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.267321    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a7eaac-68fd486df8-vfdw2"]
Sep 11 04:12:40.612264 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:40.612170    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f216b913-95a5-4d2d-a187-afc430bcaae4" path="/var/lib/kubelet/pods/f216b913-95a5-4d2d-a187-afc430bcaae4/volumes"
Sep 11 04:12:43.584270 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:12:43.584237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:13:02.455019 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.454987    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"]
Sep 11 04:13:02.457700 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.455228    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9" podUID="37b71b87-d7f2-4a70-a494-deba43f36d29" containerName="payload-processing" containerID="cri-o://f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a" gracePeriod=30
Sep 11 04:13:02.461169 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.461146    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"]
Sep 11 04:13:02.461394 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.461324    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz" podUID="a7822b86-770b-49d9-8554-e6b16c8d31f7" containerName="payload-pre-processing" containerID="cri-o://7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a" gracePeriod=30
Sep 11 04:13:02.747274 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.747248    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:13:02.749759 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.749738    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:13:02.886594 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.886567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6qjd\" (UniqueName: \"kubernetes.io/projected/37b71b87-d7f2-4a70-a494-deba43f36d29-kube-api-access-k6qjd\") pod \"37b71b87-d7f2-4a70-a494-deba43f36d29\" (UID: \"37b71b87-d7f2-4a70-a494-deba43f36d29\") "
Sep 11 04:13:02.886754 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.886683    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7822b86-770b-49d9-8554-e6b16c8d31f7-plugins-config-volume\") pod \"a7822b86-770b-49d9-8554-e6b16c8d31f7\" (UID: \"a7822b86-770b-49d9-8554-e6b16c8d31f7\") "
Sep 11 04:13:02.886754 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.886704    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sl2x\" (UniqueName: \"kubernetes.io/projected/a7822b86-770b-49d9-8554-e6b16c8d31f7-kube-api-access-9sl2x\") pod \"a7822b86-770b-49d9-8554-e6b16c8d31f7\" (UID: \"a7822b86-770b-49d9-8554-e6b16c8d31f7\") "
Sep 11 04:13:02.886754 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.886732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b71b87-d7f2-4a70-a494-deba43f36d29-plugins-config-volume\") pod \"37b71b87-d7f2-4a70-a494-deba43f36d29\" (UID: \"37b71b87-d7f2-4a70-a494-deba43f36d29\") "
Sep 11 04:13:02.887104 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.887076    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7822b86-770b-49d9-8554-e6b16c8d31f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7822b86-770b-49d9-8554-e6b16c8d31f7" (UID: "a7822b86-770b-49d9-8554-e6b16c8d31f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:13:02.887250 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.887153    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b71b87-d7f2-4a70-a494-deba43f36d29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37b71b87-d7f2-4a70-a494-deba43f36d29" (UID: "37b71b87-d7f2-4a70-a494-deba43f36d29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:13:02.888729 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.888705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7822b86-770b-49d9-8554-e6b16c8d31f7-kube-api-access-9sl2x" (OuterVolumeSpecName: "kube-api-access-9sl2x") pod "a7822b86-770b-49d9-8554-e6b16c8d31f7" (UID: "a7822b86-770b-49d9-8554-e6b16c8d31f7"). InnerVolumeSpecName "kube-api-access-9sl2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:02.888843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.888746    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b71b87-d7f2-4a70-a494-deba43f36d29-kube-api-access-k6qjd" (OuterVolumeSpecName: "kube-api-access-k6qjd") pod "37b71b87-d7f2-4a70-a494-deba43f36d29" (UID: "37b71b87-d7f2-4a70-a494-deba43f36d29"). InnerVolumeSpecName "kube-api-access-k6qjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:02.988331 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.988272    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7822b86-770b-49d9-8554-e6b16c8d31f7-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:02.988331 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.988300    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sl2x\" (UniqueName: \"kubernetes.io/projected/a7822b86-770b-49d9-8554-e6b16c8d31f7-kube-api-access-9sl2x\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:02.988331 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.988314    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b71b87-d7f2-4a70-a494-deba43f36d29-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:02.988331 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:02.988327    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qjd\" (UniqueName: \"kubernetes.io/projected/37b71b87-d7f2-4a70-a494-deba43f36d29-kube-api-access-k6qjd\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:03.081174 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.081143    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:13:03.351738 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.351655    2566 generic.go:356] "Generic (PLEG): container finished" podID="a7822b86-770b-49d9-8554-e6b16c8d31f7" containerID="7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a" exitCode=0
Sep 11 04:13:03.351738 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.351693    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"
Sep 11 04:13:03.351970 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.351736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz" event={"ID":"a7822b86-770b-49d9-8554-e6b16c8d31f7","Type":"ContainerDied","Data":"7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a"}
Sep 11 04:13:03.351970 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.351785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz" event={"ID":"a7822b86-770b-49d9-8554-e6b16c8d31f7","Type":"ContainerDied","Data":"c19b2cabc1e1cf352585750dc6ef5e00ded083217e1084ac52c144f069c43de4"}
Sep 11 04:13:03.351970 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.351824    2566 scope.go:117] "RemoveContainer" containerID="7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a"
Sep 11 04:13:03.353292 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.353268    2566 generic.go:356] "Generic (PLEG): container finished" podID="37b71b87-d7f2-4a70-a494-deba43f36d29" containerID="f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a" exitCode=0
Sep 11 04:13:03.353399 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.353303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9" event={"ID":"37b71b87-d7f2-4a70-a494-deba43f36d29","Type":"ContainerDied","Data":"f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a"}
Sep 11 04:13:03.353399 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.353310    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"
Sep 11 04:13:03.353399 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.353344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9" event={"ID":"37b71b87-d7f2-4a70-a494-deba43f36d29","Type":"ContainerDied","Data":"eeb7d7c061aacbdf7f956d943578e9fe94c90bb7d3dafa9419454adfcc859f99"}
Sep 11 04:13:03.361505 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.361488    2566 scope.go:117] "RemoveContainer" containerID="7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a"
Sep 11 04:13:03.361793 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:13:03.361772    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a\": container with ID starting with 7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a not found: ID does not exist" containerID="7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a"
Sep 11 04:13:03.361853 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.361801    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a"} err="failed to get container status \"7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a\": rpc error: code = NotFound desc = could not find container \"7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a\": container with ID starting with 7a7e7f1e1dddedd5aac2d03443b8ae5fc439dffc6a4cb074172cd8d33f9d135a not found: ID does not exist"
Sep 11 04:13:03.361853 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.361817    2566 scope.go:117] "RemoveContainer" containerID="f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a"
Sep 11 04:13:03.369193 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.369174    2566 scope.go:117] "RemoveContainer" containerID="f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a"
Sep 11 04:13:03.369450 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:13:03.369427    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a\": container with ID starting with f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a not found: ID does not exist" containerID="f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a"
Sep 11 04:13:03.369592 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.369459    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a"} err="failed to get container status \"f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a\": rpc error: code = NotFound desc = could not find container \"f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a\": container with ID starting with f578e9e6e5f7507c3dd5435e96f53c59253b0aa39d506291a9b5a147eb406a4a not found: ID does not exist"
Sep 11 04:13:03.373896 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.373873    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"]
Sep 11 04:13:03.377510 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.377492    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4c6f83-7c44947bc8-thqqz"]
Sep 11 04:13:03.387442 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.387397    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"]
Sep 11 04:13:03.389321 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:03.389302    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4c6f83-5f8976cf65-m5mq9"]
Sep 11 04:13:04.611969 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:04.611939    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b71b87-d7f2-4a70-a494-deba43f36d29" path="/var/lib/kubelet/pods/37b71b87-d7f2-4a70-a494-deba43f36d29/volumes"
Sep 11 04:13:04.612336 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:04.612288    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7822b86-770b-49d9-8554-e6b16c8d31f7" path="/var/lib/kubelet/pods/a7822b86-770b-49d9-8554-e6b16c8d31f7/volumes"
Sep 11 04:13:14.979400 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:14.979369    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:13:20.200291 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.200254    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"]
Sep 11 04:13:20.200788 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.200522    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb" podUID="4512b40e-1e41-49fc-a92e-e888bbf9dee1" containerName="payload-pre-processing" containerID="cri-o://e70c521054a1f0e2dc18d745816a7a14cb41bde68bc0795f901919f7768f8375" gracePeriod=30
Sep 11 04:13:20.439913 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.439880    2566 generic.go:356] "Generic (PLEG): container finished" podID="4512b40e-1e41-49fc-a92e-e888bbf9dee1" containerID="e70c521054a1f0e2dc18d745816a7a14cb41bde68bc0795f901919f7768f8375" exitCode=0
Sep 11 04:13:20.440044 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.439937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb" event={"ID":"4512b40e-1e41-49fc-a92e-e888bbf9dee1","Type":"ContainerDied","Data":"e70c521054a1f0e2dc18d745816a7a14cb41bde68bc0795f901919f7768f8375"}
Sep 11 04:13:20.458500 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.458447    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:13:20.638581 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.638554    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4512b40e-1e41-49fc-a92e-e888bbf9dee1-plugins-config-volume\") pod \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\" (UID: \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\") "
Sep 11 04:13:20.638732 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.638642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5jff\" (UniqueName: \"kubernetes.io/projected/4512b40e-1e41-49fc-a92e-e888bbf9dee1-kube-api-access-s5jff\") pod \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\" (UID: \"4512b40e-1e41-49fc-a92e-e888bbf9dee1\") "
Sep 11 04:13:20.638970 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.638945    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4512b40e-1e41-49fc-a92e-e888bbf9dee1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4512b40e-1e41-49fc-a92e-e888bbf9dee1" (UID: "4512b40e-1e41-49fc-a92e-e888bbf9dee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:13:20.640630 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.640605    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4512b40e-1e41-49fc-a92e-e888bbf9dee1-kube-api-access-s5jff" (OuterVolumeSpecName: "kube-api-access-s5jff") pod "4512b40e-1e41-49fc-a92e-e888bbf9dee1" (UID: "4512b40e-1e41-49fc-a92e-e888bbf9dee1"). InnerVolumeSpecName "kube-api-access-s5jff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:20.740129 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.740105    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4512b40e-1e41-49fc-a92e-e888bbf9dee1-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:20.740129 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:20.740131    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jff\" (UniqueName: \"kubernetes.io/projected/4512b40e-1e41-49fc-a92e-e888bbf9dee1-kube-api-access-s5jff\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:21.446626 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:21.446593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb" event={"ID":"4512b40e-1e41-49fc-a92e-e888bbf9dee1","Type":"ContainerDied","Data":"957d450d1077c2b432616f7700683d6bc1d68884b559bdf8339489c494d2eb1d"}
Sep 11 04:13:21.446626 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:21.446614    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"
Sep 11 04:13:21.447137 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:21.446660    2566 scope.go:117] "RemoveContainer" containerID="e70c521054a1f0e2dc18d745816a7a14cb41bde68bc0795f901919f7768f8375"
Sep 11 04:13:21.468482 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:21.468459    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"]
Sep 11 04:13:21.475281 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:21.475260    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-68ec26-fc57dc9c8-xgqjb"]
Sep 11 04:13:22.612728 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:22.612694    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4512b40e-1e41-49fc-a92e-e888bbf9dee1" path="/var/lib/kubelet/pods/4512b40e-1e41-49fc-a92e-e888bbf9dee1/volumes"
Sep 11 04:13:27.415354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.415323    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:13:27.422742 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.422717    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"]
Sep 11 04:13:27.422931 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.422910    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy" containerID="cri-o://5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933" gracePeriod=30
Sep 11 04:13:27.537973 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.537947    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"]
Sep 11 04:13:27.538302 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538291    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4512b40e-1e41-49fc-a92e-e888bbf9dee1" containerName="payload-pre-processing"
Sep 11 04:13:27.538354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538304    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4512b40e-1e41-49fc-a92e-e888bbf9dee1" containerName="payload-pre-processing"
Sep 11 04:13:27.538354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538312    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b71b87-d7f2-4a70-a494-deba43f36d29" containerName="payload-processing"
Sep 11 04:13:27.538354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538318    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b71b87-d7f2-4a70-a494-deba43f36d29" containerName="payload-processing"
Sep 11 04:13:27.538354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538327    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7822b86-770b-49d9-8554-e6b16c8d31f7" containerName="payload-pre-processing"
Sep 11 04:13:27.538354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538332    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7822b86-770b-49d9-8554-e6b16c8d31f7" containerName="payload-pre-processing"
Sep 11 04:13:27.538354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538345    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f216b913-95a5-4d2d-a187-afc430bcaae4" containerName="payload-pre-processing"
Sep 11 04:13:27.538354 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538351    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f216b913-95a5-4d2d-a187-afc430bcaae4" containerName="payload-pre-processing"
Sep 11 04:13:27.538610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538365    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348bf11c-6015-4afb-ae3c-aec6c3ce02e1" containerName="main"
Sep 11 04:13:27.538610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538370    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="348bf11c-6015-4afb-ae3c-aec6c3ce02e1" containerName="main"
Sep 11 04:13:27.538610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538432    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f216b913-95a5-4d2d-a187-afc430bcaae4" containerName="payload-pre-processing"
Sep 11 04:13:27.538610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538441    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="348bf11c-6015-4afb-ae3c-aec6c3ce02e1" containerName="main"
Sep 11 04:13:27.538610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538448    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b71b87-d7f2-4a70-a494-deba43f36d29" containerName="payload-processing"
Sep 11 04:13:27.538610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538454    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7822b86-770b-49d9-8554-e6b16c8d31f7" containerName="payload-pre-processing"
Sep 11 04:13:27.538610 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.538459    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4512b40e-1e41-49fc-a92e-e888bbf9dee1" containerName="payload-pre-processing"
Sep 11 04:13:27.542746 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.542729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:27.545459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.545438    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ef4b62\""
Sep 11 04:13:27.553262 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.553240    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"]
Sep 11 04:13:27.592225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.592196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hmg\" (UniqueName: \"kubernetes.io/projected/584ecaf6-96e0-4937-ac48-c3008904cfe5-kube-api-access-h5hmg\") pod \"payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj\" (UID: \"584ecaf6-96e0-4937-ac48-c3008904cfe5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:27.592347 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.592240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584ecaf6-96e0-4937-ac48-c3008904cfe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj\" (UID: \"584ecaf6-96e0-4937-ac48-c3008904cfe5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:27.693381 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.693300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hmg\" (UniqueName: \"kubernetes.io/projected/584ecaf6-96e0-4937-ac48-c3008904cfe5-kube-api-access-h5hmg\") pod \"payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj\" (UID: \"584ecaf6-96e0-4937-ac48-c3008904cfe5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:27.693381 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.693362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584ecaf6-96e0-4937-ac48-c3008904cfe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj\" (UID: \"584ecaf6-96e0-4937-ac48-c3008904cfe5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:27.694101 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.694075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584ecaf6-96e0-4937-ac48-c3008904cfe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj\" (UID: \"584ecaf6-96e0-4937-ac48-c3008904cfe5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:27.706376 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.706344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hmg\" (UniqueName: \"kubernetes.io/projected/584ecaf6-96e0-4937-ac48-c3008904cfe5-kube-api-access-h5hmg\") pod \"payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj\" (UID: \"584ecaf6-96e0-4937-ac48-c3008904cfe5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:27.853116 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:27.853081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:28.153927 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:28.153904    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"]
Sep 11 04:13:28.156210 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:13:28.156182    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584ecaf6_96e0_4937_ac48_c3008904cfe5.slice/crio-169ac1859e1ba7064cd9e10ba8208d78e7feaa992457ba98d65fa5714daaf883 WatchSource:0}: Error finding container 169ac1859e1ba7064cd9e10ba8208d78e7feaa992457ba98d65fa5714daaf883: Status 404 returned error can't find the container with id 169ac1859e1ba7064cd9e10ba8208d78e7feaa992457ba98d65fa5714daaf883
Sep 11 04:13:28.483840 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:28.483801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj" event={"ID":"584ecaf6-96e0-4937-ac48-c3008904cfe5","Type":"ContainerStarted","Data":"fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14"}
Sep 11 04:13:28.484218 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:28.483867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj" event={"ID":"584ecaf6-96e0-4937-ac48-c3008904cfe5","Type":"ContainerStarted","Data":"169ac1859e1ba7064cd9e10ba8208d78e7feaa992457ba98d65fa5714daaf883"}
Sep 11 04:13:28.484218 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:28.483897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:28.503368 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:28.503323    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj" podStartSLOduration=1.50330927 podStartE2EDuration="1.50330927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:28.501178377 +0000 UTC m=+1406.367198111" watchObservedRunningTime="2026-09-11 04:13:28.50330927 +0000 UTC m=+1406.369329049"
Sep 11 04:13:32.668841 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.668816    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:13:32.743802 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.743769    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-token\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.743926 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.743820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-podinfo\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.743926 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.743857    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istiod-ca-cert\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.743926 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.743883    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-socket\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.744103 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.743978    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrv2r\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-kube-api-access-vrv2r\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.744103 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-credential-socket\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.744103 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-certs\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.744103 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744103    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-envoy\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.744309 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744138    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-data\") pod \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\" (UID: \"a2c9907f-aeab-47fe-83a9-12940ecd7da6\") "
Sep 11 04:13:32.744374 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:32.744374 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744325    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:32.744374 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744340    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:13:32.744555 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744532    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:32.744607 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744556    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:32.744607 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744576    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:32.744607 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744591    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:32.744729 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.744641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:32.745837 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.745817    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:32.746122 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.746108    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:32.746278 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.746257    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-kube-api-access-vrv2r" (OuterVolumeSpecName: "kube-api-access-vrv2r") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "kube-api-access-vrv2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:32.746278 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.746271    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2c9907f-aeab-47fe-83a9-12940ecd7da6" (UID: "a2c9907f-aeab-47fe-83a9-12940ecd7da6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:13:32.845806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.845784    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:32.845806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.845802    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:32.845928 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.845811    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:32.845928 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.845820    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrv2r\" (UniqueName: \"kubernetes.io/projected/a2c9907f-aeab-47fe-83a9-12940ecd7da6-kube-api-access-vrv2r\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:32.845928 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.845828    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:32.845928 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:32.845838    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c9907f-aeab-47fe-83a9-12940ecd7da6-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:33.507666 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.507635    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerID="5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933" exitCode=0
Sep 11 04:13:33.507876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.507684    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"
Sep 11 04:13:33.507876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.507714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" event={"ID":"a2c9907f-aeab-47fe-83a9-12940ecd7da6","Type":"ContainerDied","Data":"5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933"}
Sep 11 04:13:33.507876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.507768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r" event={"ID":"a2c9907f-aeab-47fe-83a9-12940ecd7da6","Type":"ContainerDied","Data":"9d51c1ac1aa353d609c0c06cf929b31232958dd5d8dbdb4a11de5d3b85f30d06"}
Sep 11 04:13:33.507876 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.507796    2566 scope.go:117] "RemoveContainer" containerID="5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933"
Sep 11 04:13:33.517320 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.517305    2566 scope.go:117] "RemoveContainer" containerID="38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766"
Sep 11 04:13:33.525220 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.525206    2566 scope.go:117] "RemoveContainer" containerID="5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933"
Sep 11 04:13:33.525487 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:13:33.525469    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933\": container with ID starting with 5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933 not found: ID does not exist" containerID="5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933"
Sep 11 04:13:33.525531 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.525495    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933"} err="failed to get container status \"5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933\": rpc error: code = NotFound desc = could not find container \"5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933\": container with ID starting with 5787ae9e563e42261bd0519e5ad8b8aae69a165c2f4caa1b8ed53fd9bcb18933 not found: ID does not exist"
Sep 11 04:13:33.525531 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.525510    2566 scope.go:117] "RemoveContainer" containerID="38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766"
Sep 11 04:13:33.525722 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:13:33.525710    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766\": container with ID starting with 38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766 not found: ID does not exist" containerID="38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766"
Sep 11 04:13:33.525763 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.525726    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766"} err="failed to get container status \"38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766\": rpc error: code = NotFound desc = could not find container \"38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766\": container with ID starting with 38bf4e9be230887bd41b9568d3819d31bdee8350fb9e8d5843abfee983405766 not found: ID does not exist"
Sep 11 04:13:33.541104 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.541083    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"]
Sep 11 04:13:33.547173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:33.547154    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-68ec26-openshift-default-7dc785bff8-c2v7r"]
Sep 11 04:13:34.611704 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:34.611669    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" path="/var/lib/kubelet/pods/a2c9907f-aeab-47fe-83a9-12940ecd7da6/volumes"
Sep 11 04:13:39.399958 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:39.399918    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:13:39.490597 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:39.490564    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:13:44.986956 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:44.986916    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"]
Sep 11 04:13:44.987490 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:44.987116    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy" containerID="cri-o://9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a" gracePeriod=30
Sep 11 04:13:46.189164 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:46.189117    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 04:13:46.189570 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:46.189172    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 04:13:47.688024 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:47.687989    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:13:50.242522 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.242502    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:13:50.399986 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.399911    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-data\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.399986 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.399967    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9k8b\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-kube-api-access-j9k8b\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.400190 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.399996    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-certs\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.400190 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-istio-token\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.400190 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-envoy\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.400190 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48989c5-5227-4919-87ad-414b2294e431-istiod-ca-cert\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.400190 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-socket\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.400190 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400143    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48989c5-5227-4919-87ad-414b2294e431-istio-podinfo\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.400551 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400194    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-data" (OuterVolumeSpecName: "istio-data") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:50.400551 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400204    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-credential-socket\") pod \"e48989c5-5227-4919-87ad-414b2294e431\" (UID: \"e48989c5-5227-4919-87ad-414b2294e431\") "
Sep 11 04:13:50.400551 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400265    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:50.400551 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:50.400720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48989c5-5227-4919-87ad-414b2294e431-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:13:50.400720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400633    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.400720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400652    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.400720 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400665    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.400906 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.400886    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:50.402324 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.402304    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e48989c5-5227-4919-87ad-414b2294e431-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:13:50.402396 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.402343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-istio-token" (OuterVolumeSpecName: "istio-token") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:50.402640 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.402613    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-kube-api-access-j9k8b" (OuterVolumeSpecName: "kube-api-access-j9k8b") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "kube-api-access-j9k8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:13:50.402640 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.402613    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e48989c5-5227-4919-87ad-414b2294e431" (UID: "e48989c5-5227-4919-87ad-414b2294e431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:13:50.501480 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.501450    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9k8b\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-kube-api-access-j9k8b\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.501480 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.501474    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48989c5-5227-4919-87ad-414b2294e431-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.501480 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.501483    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.501695 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.501491    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48989c5-5227-4919-87ad-414b2294e431-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.501695 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.501501    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48989c5-5227-4919-87ad-414b2294e431-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.501695 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.501509    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48989c5-5227-4919-87ad-414b2294e431-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:13:50.599083 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.599052    2566 generic.go:356] "Generic (PLEG): container finished" podID="e48989c5-5227-4919-87ad-414b2294e431" containerID="9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a" exitCode=0
Sep 11 04:13:50.599221 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.599091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" event={"ID":"e48989c5-5227-4919-87ad-414b2294e431","Type":"ContainerDied","Data":"9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a"}
Sep 11 04:13:50.599221 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.599101    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"
Sep 11 04:13:50.599221 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.599143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh" event={"ID":"e48989c5-5227-4919-87ad-414b2294e431","Type":"ContainerDied","Data":"f51f731cf447c32d8f573fc29e4279efd22e9eee02d0182ac3c98ba2add804fe"}
Sep 11 04:13:50.599221 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.599165    2566 scope.go:117] "RemoveContainer" containerID="9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a"
Sep 11 04:13:50.608516 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.608495    2566 scope.go:117] "RemoveContainer" containerID="b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374"
Sep 11 04:13:50.616778 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.616764    2566 scope.go:117] "RemoveContainer" containerID="9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a"
Sep 11 04:13:50.617011 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:13:50.616994    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a\": container with ID starting with 9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a not found: ID does not exist" containerID="9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a"
Sep 11 04:13:50.617051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.617018    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a"} err="failed to get container status \"9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a\": rpc error: code = NotFound desc = could not find container \"9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a\": container with ID starting with 9b0041829418c5a48e69e0671293d975272d8b6986e756de9868278317fb571a not found: ID does not exist"
Sep 11 04:13:50.617051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.617033    2566 scope.go:117] "RemoveContainer" containerID="b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374"
Sep 11 04:13:50.617235 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:13:50.617220    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374\": container with ID starting with b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374 not found: ID does not exist" containerID="b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374"
Sep 11 04:13:50.617269 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.617239    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374"} err="failed to get container status \"b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374\": rpc error: code = NotFound desc = could not find container \"b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374\": container with ID starting with b42ff4de36e91964089edd5109b53bc8cd1dba7f475668333128ce02fb4a7374 not found: ID does not exist"
Sep 11 04:13:50.620487 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.620467    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"]
Sep 11 04:13:50.626202 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:50.626184    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-604323-openshift-default-5756576dd-kvjnh"]
Sep 11 04:13:51.400683 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.400651    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"]
Sep 11 04:13:51.401183 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401168    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy"
Sep 11 04:13:51.401230 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401188    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy"
Sep 11 04:13:51.401230 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401202    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy"
Sep 11 04:13:51.401230 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401211    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy"
Sep 11 04:13:51.401326 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401232    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy"
Sep 11 04:13:51.401326 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401240    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy"
Sep 11 04:13:51.401390 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401331    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy"
Sep 11 04:13:51.401390 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401341    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy"
Sep 11 04:13:51.401390 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401350    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy"
Sep 11 04:13:51.401390 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.401360    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48989c5-5227-4919-87ad-414b2294e431" containerName="istio-proxy"
Sep 11 04:13:51.404338 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.404320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.407236 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.407218    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-85dc29-openshift-default-dockercfg-xvhzc\""
Sep 11 04:13:51.416903 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.416882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"]
Sep 11 04:13:51.510563 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-podinfo\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.510563 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df030f2a-b43e-4155-ae88-fdbb43d924a8-istiod-ca-cert\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.510816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-credential-socket\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.510816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmb5\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-kube-api-access-qnmb5\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.510816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510739    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-socket\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.510816 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-envoy\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.510972 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-certs\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.510972 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-token\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.510972 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.510964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-data\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.611718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.611689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-podinfo\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.611718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.611721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df030f2a-b43e-4155-ae88-fdbb43d924a8-istiod-ca-cert\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.611993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.611753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-credential-socket\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.611993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.611858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmb5\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-kube-api-access-qnmb5\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.611993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.611890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-socket\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.611993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.611909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-envoy\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.611993 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.611941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-certs\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.612264 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.612012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-token\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.612264 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.612042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-data\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.612376 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.612321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-credential-socket\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.612462 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.612401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-data\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.612520 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.612495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-certs\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.612580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.612539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-socket\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.612580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.612546    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df030f2a-b43e-4155-ae88-fdbb43d924a8-istiod-ca-cert\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.614242 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.614221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-podinfo\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.614362 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.614293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-envoy\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.619518 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.619489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmb5\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-kube-api-access-qnmb5\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.619518 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.619500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-token\") pod \"e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") " pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.716438 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.716396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:51.990033 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.989955    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"]
Sep 11 04:13:51.992940 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:13:51.992911    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf030f2a_b43e_4155_ae88_fdbb43d924a8.slice/crio-b2458c4aeb7ebbf66a22d74f1a8444b61b31d2c29943f3c1e284426aa19d8c4a WatchSource:0}: Error finding container b2458c4aeb7ebbf66a22d74f1a8444b61b31d2c29943f3c1e284426aa19d8c4a: Status 404 returned error can't find the container with id b2458c4aeb7ebbf66a22d74f1a8444b61b31d2c29943f3c1e284426aa19d8c4a
Sep 11 04:13:51.995114 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.995081    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:13:51.995195 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.995144    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:13:51.995195 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:51.995168    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:13:52.612750 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:52.612719    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48989c5-5227-4919-87ad-414b2294e431" path="/var/lib/kubelet/pods/e48989c5-5227-4919-87ad-414b2294e431/volumes"
Sep 11 04:13:52.613115 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:52.612979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" event={"ID":"df030f2a-b43e-4155-ae88-fdbb43d924a8","Type":"ContainerStarted","Data":"14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f"}
Sep 11 04:13:52.613115 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:52.613015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" event={"ID":"df030f2a-b43e-4155-ae88-fdbb43d924a8","Type":"ContainerStarted","Data":"b2458c4aeb7ebbf66a22d74f1a8444b61b31d2c29943f3c1e284426aa19d8c4a"}
Sep 11 04:13:52.632868 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:52.632812    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" podStartSLOduration=1.632798378 podStartE2EDuration="1.632798378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:52.630816172 +0000 UTC m=+1430.496835933" watchObservedRunningTime="2026-09-11 04:13:52.632798378 +0000 UTC m=+1430.498818115"
Sep 11 04:13:52.717334 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:52.717289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:53.722114 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:53.722084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:54.621732 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:54.621695    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:54.622783 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:54.622763    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:13:56.303929 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.303846    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"]
Sep 11 04:13:56.304324 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.304219    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy"
Sep 11 04:13:56.304324 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.304230    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c9907f-aeab-47fe-83a9-12940ecd7da6" containerName="istio-proxy"
Sep 11 04:13:56.306230 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.306214    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:56.308599 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.308577    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-85dc29-dockercfg-fqp7n\""
Sep 11 04:13:56.308703 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.308610    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-85dc29\""
Sep 11 04:13:56.320013 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.319985    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"]
Sep 11 04:13:56.457602 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.457560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6826ca-b122-4a2e-b7d3-85c49af57968-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6\" (UID: \"bd6826ca-b122-4a2e-b7d3-85c49af57968\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:56.457756 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.457613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5csf\" (UniqueName: \"kubernetes.io/projected/bd6826ca-b122-4a2e-b7d3-85c49af57968-kube-api-access-g5csf\") pod \"payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6\" (UID: \"bd6826ca-b122-4a2e-b7d3-85c49af57968\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:56.559172 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.559090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6826ca-b122-4a2e-b7d3-85c49af57968-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6\" (UID: \"bd6826ca-b122-4a2e-b7d3-85c49af57968\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:56.559172 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.559125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5csf\" (UniqueName: \"kubernetes.io/projected/bd6826ca-b122-4a2e-b7d3-85c49af57968-kube-api-access-g5csf\") pod \"payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6\" (UID: \"bd6826ca-b122-4a2e-b7d3-85c49af57968\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:56.559866 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.559844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6826ca-b122-4a2e-b7d3-85c49af57968-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6\" (UID: \"bd6826ca-b122-4a2e-b7d3-85c49af57968\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:56.567463 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.567434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5csf\" (UniqueName: \"kubernetes.io/projected/bd6826ca-b122-4a2e-b7d3-85c49af57968-kube-api-access-g5csf\") pod \"payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6\" (UID: \"bd6826ca-b122-4a2e-b7d3-85c49af57968\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:56.616389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.616358    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:56.890345 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:56.890322    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"]
Sep 11 04:13:56.892375 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:13:56.892346    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6826ca_b122_4a2e_b7d3_85c49af57968.slice/crio-fba0efab65acea385d067e05c6541c8f90af5f02bb1435cf29457b79573a721c WatchSource:0}: Error finding container fba0efab65acea385d067e05c6541c8f90af5f02bb1435cf29457b79573a721c: Status 404 returned error can't find the container with id fba0efab65acea385d067e05c6541c8f90af5f02bb1435cf29457b79573a721c
Sep 11 04:13:57.639101 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:57.639061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6" event={"ID":"bd6826ca-b122-4a2e-b7d3-85c49af57968","Type":"ContainerStarted","Data":"c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee"}
Sep 11 04:13:57.639517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:57.639117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6" event={"ID":"bd6826ca-b122-4a2e-b7d3-85c49af57968","Type":"ContainerStarted","Data":"fba0efab65acea385d067e05c6541c8f90af5f02bb1435cf29457b79573a721c"}
Sep 11 04:13:57.639517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:57.639142    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:13:57.657339 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:13:57.657287    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6" podStartSLOduration=1.657274398 podStartE2EDuration="1.657274398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:57.654739415 +0000 UTC m=+1435.520759154" watchObservedRunningTime="2026-09-11 04:13:57.657274398 +0000 UTC m=+1435.523294135"
Sep 11 04:14:02.388197 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:02.388168    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:14:08.646614 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:08.646574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:14:13.717710 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:13.717675    2566 generic.go:356] "Generic (PLEG): container finished" podID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerID="14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f" exitCode=0
Sep 11 04:14:13.718243 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:13.717735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" event={"ID":"df030f2a-b43e-4155-ae88-fdbb43d924a8","Type":"ContainerDied","Data":"14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f"}
Sep 11 04:14:13.718243 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:13.718192    2566 scope.go:117] "RemoveContainer" containerID="14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f"
Sep 11 04:14:13.719294 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:13.719262    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:14:13.719386 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:13.719333    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:14:13.719386 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:13.719365    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 04:14:14.724558 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:14.724517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" event={"ID":"df030f2a-b43e-4155-ae88-fdbb43d924a8","Type":"ContainerStarted","Data":"0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d"}
Sep 11 04:14:15.716842 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:15.716802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:14:15.717126 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:15.717106    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 11 04:14:15.717176 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:15.717154    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 11 04:14:16.716760 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:16.716720    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 11 04:14:16.717209 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:16.716803    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 11 04:14:17.717289 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:17.717244    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 11 04:14:17.717800 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:17.717310    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 11 04:14:18.717336 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:18.717301    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 11 04:14:18.717833 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:18.717395    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 11 04:14:19.717044 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:19.716997    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 11 04:14:19.717232 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:19.717066    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 11 04:14:20.720651 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:20.720623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:14:20.721011 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:20.720898    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:14:20.721599 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:20.721579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:14:24.351750 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:24.351716    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l5kqq_3338e8c2-2273-4099-aed2-4a55c00c007e/payload-processing/0.log"
Sep 11 04:14:24.462252 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:24.462221    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6_bd6826ca-b122-4a2e-b7d3-85c49af57968/payload-processing/0.log"
Sep 11 04:14:36.217392 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.217360    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"]
Sep 11 04:14:36.220820 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.220803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.223548 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.223525    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ef4b62\"/\"kube-root-ca.crt\""
Sep 11 04:14:36.224683 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.224668    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ef4b62\"/\"openshift-service-ca.crt\""
Sep 11 04:14:36.224776 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.224714    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ef4b62\"/\"default-dockercfg-q2d58\""
Sep 11 04:14:36.224931 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.224916    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ef4b62\"/\"ipp-route-ef4b62-kserve-self-signed-certs\""
Sep 11 04:14:36.231343 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.231320    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"]
Sep 11 04:14:36.351777 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.351745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86zp\" (UniqueName: \"kubernetes.io/projected/32c032ae-a660-45fa-9876-032e7c320ff3-kube-api-access-s86zp\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.351961 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.351799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c032ae-a660-45fa-9876-032e7c320ff3-tls-certs\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.351961 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.351822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-tmp-dir\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.351961 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.351909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-model-cache\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.352076 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.351987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-home\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.352076 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.352002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-dshm\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453268 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-model-cache\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453458 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-home\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453458 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-dshm\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453458 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86zp\" (UniqueName: \"kubernetes.io/projected/32c032ae-a660-45fa-9876-032e7c320ff3-kube-api-access-s86zp\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453458 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c032ae-a660-45fa-9876-032e7c320ff3-tls-certs\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453458 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-tmp-dir\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453742 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-model-cache\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453800 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-home\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.453847 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.453803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-tmp-dir\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.455727 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.455696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-dshm\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.455927 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.455910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c032ae-a660-45fa-9876-032e7c320ff3-tls-certs\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.463963 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.463943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86zp\" (UniqueName: \"kubernetes.io/projected/32c032ae-a660-45fa-9876-032e7c320ff3-kube-api-access-s86zp\") pod \"ipp-route-ef4b62-kserve-548c545f87-7czn8\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") " pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.530440 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.530347    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:36.805628 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.805605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"]
Sep 11 04:14:36.807438 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:14:36.807396    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32c032ae_a660_45fa_9876_032e7c320ff3.slice/crio-04bc29539d06fd8b3e1daff3f0539ba6188bef76c01b603482d57432467a2a87 WatchSource:0}: Error finding container 04bc29539d06fd8b3e1daff3f0539ba6188bef76c01b603482d57432467a2a87: Status 404 returned error can't find the container with id 04bc29539d06fd8b3e1daff3f0539ba6188bef76c01b603482d57432467a2a87
Sep 11 04:14:36.833188 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:36.833163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8" event={"ID":"32c032ae-a660-45fa-9876-032e7c320ff3","Type":"ContainerStarted","Data":"04bc29539d06fd8b3e1daff3f0539ba6188bef76c01b603482d57432467a2a87"}
Sep 11 04:14:37.840054 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:37.840015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8" event={"ID":"32c032ae-a660-45fa-9876-032e7c320ff3","Type":"ContainerStarted","Data":"a8de040f045f29986dc1a263ceb5b245a9fe6e7ce94c62cc1a87ab09053a3fac"}
Sep 11 04:14:37.858731 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:37.858683    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8" podStartSLOduration=1.85866831 podStartE2EDuration="1.85866831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:37.856371755 +0000 UTC m=+1475.722391490" watchObservedRunningTime="2026-09-11 04:14:37.85866831 +0000 UTC m=+1475.724688095"
Sep 11 04:14:40.186558 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:40.186524    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:14:46.531295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:46.531246    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:46.531295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:46.531295    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:46.544295 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:46.544260    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:14:46.893680 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:14:46.893592    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:15:00.138756 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.138721    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb"]
Sep 11 04:15:00.142763 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.142740    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"]
Sep 11 04:15:00.142951 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.142861    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb"
Sep 11 04:15:00.145630 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.145598    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wbtxz\""
Sep 11 04:15:00.146105 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.146086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"
Sep 11 04:15:00.157021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.156774    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb"
Sep 11 04:15:00.159279 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.159207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"]
Sep 11 04:15:00.163307 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.162696    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb"]
Sep 11 04:15:00.165213 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.165194    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"
Sep 11 04:15:00.436000 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.435975    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb"]
Sep 11 04:15:00.438189 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:15:00.438153    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c742d05_f1db_4f31_8a5f_e491eb01e4cd.slice/crio-6a6a0fd66b7f15cadb909bc354dbf700fde3fd636f666e0c8a20dee36c5bf734 WatchSource:0}: Error finding container 6a6a0fd66b7f15cadb909bc354dbf700fde3fd636f666e0c8a20dee36c5bf734: Status 404 returned error can't find the container with id 6a6a0fd66b7f15cadb909bc354dbf700fde3fd636f666e0c8a20dee36c5bf734
Sep 11 04:15:00.440120 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.440104    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:15:00.448866 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.448831    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"]
Sep 11 04:15:00.449671 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:15:00.449640    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7474158b_b28a_498e_9e58_a1ca58fd79ae.slice/crio-e1a1a39e7bc746a5995298986816b3e95a87d328dd53f83d66ff4146262c529b WatchSource:0}: Error finding container e1a1a39e7bc746a5995298986816b3e95a87d328dd53f83d66ff4146262c529b: Status 404 returned error can't find the container with id e1a1a39e7bc746a5995298986816b3e95a87d328dd53f83d66ff4146262c529b
Sep 11 04:15:00.957359 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.957323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb" event={"ID":"7474158b-b28a-498e-9e58-a1ca58fd79ae","Type":"ContainerStarted","Data":"e1a1a39e7bc746a5995298986816b3e95a87d328dd53f83d66ff4146262c529b"}
Sep 11 04:15:00.958868 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:00.958840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb" event={"ID":"0c742d05-f1db-4f31-8a5f-e491eb01e4cd","Type":"ContainerStarted","Data":"6a6a0fd66b7f15cadb909bc354dbf700fde3fd636f666e0c8a20dee36c5bf734"}
Sep 11 04:15:02.693518 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:02.693491    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:15:02.694031 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:02.693548    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:15:02.978152 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:02.978114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb" event={"ID":"7474158b-b28a-498e-9e58-a1ca58fd79ae","Type":"ContainerStarted","Data":"3ef74651b0ceaa530d6e70dd8bc5fddda63842823b80ee51e0948f01bbcbf6e5"}
Sep 11 04:15:02.991471 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:02.991444    2566 scope.go:117] "RemoveContainer" containerID="f09719ad226a35ce01d4c6c53e549aa58521c9bf04026423aee01cd69bb7689d"
Sep 11 04:15:02.994495 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:02.994439    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb" podStartSLOduration=0.690784814 podStartE2EDuration="2.994420312s" totalImagesPullingTime="2.303635498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:02.992366526 +0000 UTC m=+1500.858386266" watchObservedRunningTime="2026-09-11 04:15:02.994420312 +0000 UTC m=+1500.860440039"
Sep 11 04:15:03.977124 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:03.977087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:15:03.985096 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:03.985061    2566 generic.go:356] "Generic (PLEG): container finished" podID="7474158b-b28a-498e-9e58-a1ca58fd79ae" containerID="3ef74651b0ceaa530d6e70dd8bc5fddda63842823b80ee51e0948f01bbcbf6e5" exitCode=0
Sep 11 04:15:03.985280 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:03.985095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb" event={"ID":"7474158b-b28a-498e-9e58-a1ca58fd79ae","Type":"ContainerDied","Data":"3ef74651b0ceaa530d6e70dd8bc5fddda63842823b80ee51e0948f01bbcbf6e5"}
Sep 11 04:15:03.986900 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:03.986874    2566 generic.go:356] "Generic (PLEG): container finished" podID="0c742d05-f1db-4f31-8a5f-e491eb01e4cd" containerID="2fab3364c8a7e85a99e38b63f4f73a3faf64b65c589176105aa6ffa2da57b27d" exitCode=0
Sep 11 04:15:03.987021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:03.986960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb" event={"ID":"0c742d05-f1db-4f31-8a5f-e491eb01e4cd","Type":"ContainerDied","Data":"2fab3364c8a7e85a99e38b63f4f73a3faf64b65c589176105aa6ffa2da57b27d"}
Sep 11 04:15:05.136082 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:05.136057    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb"
Sep 11 04:15:05.165272 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:05.165243    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"
Sep 11 04:15:05.997576 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:05.997543    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"
Sep 11 04:15:05.997766 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:05.997543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb" event={"ID":"7474158b-b28a-498e-9e58-a1ca58fd79ae","Type":"ContainerDied","Data":"e1a1a39e7bc746a5995298986816b3e95a87d328dd53f83d66ff4146262c529b"}
Sep 11 04:15:05.997766 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:05.997679    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a1a39e7bc746a5995298986816b3e95a87d328dd53f83d66ff4146262c529b"
Sep 11 04:15:05.998925 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:05.998905    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb"
Sep 11 04:15:05.999058 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:05.998946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818335-nn6bb" event={"ID":"0c742d05-f1db-4f31-8a5f-e491eb01e4cd","Type":"ContainerDied","Data":"6a6a0fd66b7f15cadb909bc354dbf700fde3fd636f666e0c8a20dee36c5bf734"}
Sep 11 04:15:05.999058 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:05.998991    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a6a0fd66b7f15cadb909bc354dbf700fde3fd636f666e0c8a20dee36c5bf734"
Sep 11 04:15:10.403237 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:10.403208    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l5kqq_3338e8c2-2273-4099-aed2-4a55c00c007e/payload-processing/0.log"
Sep 11 04:15:10.515623 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:10.515597    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6_bd6826ca-b122-4a2e-b7d3-85c49af57968/payload-processing/0.log"
Sep 11 04:15:16.284244 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.284208    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"]
Sep 11 04:15:16.284644 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.284602    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7474158b-b28a-498e-9e58-a1ca58fd79ae" containerName="cleanup"
Sep 11 04:15:16.284644 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.284614    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7474158b-b28a-498e-9e58-a1ca58fd79ae" containerName="cleanup"
Sep 11 04:15:16.284644 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.284633    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c742d05-f1db-4f31-8a5f-e491eb01e4cd" containerName="cleanup"
Sep 11 04:15:16.284644 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.284638    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c742d05-f1db-4f31-8a5f-e491eb01e4cd" containerName="cleanup"
Sep 11 04:15:16.284783 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.284699    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7474158b-b28a-498e-9e58-a1ca58fd79ae" containerName="cleanup"
Sep 11 04:15:16.284783 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.284707    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c742d05-f1db-4f31-8a5f-e491eb01e4cd" containerName="cleanup"
Sep 11 04:15:16.288061 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.288043    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:16.291460 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.291432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-27c12d-dockercfg-c8fws\""
Sep 11 04:15:16.291644 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.291622    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-27c12d\""
Sep 11 04:15:16.297394 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.297372    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"]
Sep 11 04:15:16.440765 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.440726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8sb\" (UniqueName: \"kubernetes.io/projected/d209c456-cee2-4962-a671-d7c56c855aec-kube-api-access-7p8sb\") pod \"payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86\" (UID: \"d209c456-cee2-4962-a671-d7c56c855aec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:16.440938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.440907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d209c456-cee2-4962-a671-d7c56c855aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86\" (UID: \"d209c456-cee2-4962-a671-d7c56c855aec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:16.542718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.542630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d209c456-cee2-4962-a671-d7c56c855aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86\" (UID: \"d209c456-cee2-4962-a671-d7c56c855aec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:16.542718 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.542681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8sb\" (UniqueName: \"kubernetes.io/projected/d209c456-cee2-4962-a671-d7c56c855aec-kube-api-access-7p8sb\") pod \"payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86\" (UID: \"d209c456-cee2-4962-a671-d7c56c855aec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:16.543381 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.543360    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d209c456-cee2-4962-a671-d7c56c855aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86\" (UID: \"d209c456-cee2-4962-a671-d7c56c855aec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:16.552268 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.552245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8sb\" (UniqueName: \"kubernetes.io/projected/d209c456-cee2-4962-a671-d7c56c855aec-kube-api-access-7p8sb\") pod \"payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86\" (UID: \"d209c456-cee2-4962-a671-d7c56c855aec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:16.598598 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.598560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:16.875869 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:16.875824    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"]
Sep 11 04:15:16.877964 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:15:16.877934    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd209c456_cee2_4962_a671_d7c56c855aec.slice/crio-1d454e98d98181f39521e67767e0ae042561ef2b6190e35341e3a608c77660f0 WatchSource:0}: Error finding container 1d454e98d98181f39521e67767e0ae042561ef2b6190e35341e3a608c77660f0: Status 404 returned error can't find the container with id 1d454e98d98181f39521e67767e0ae042561ef2b6190e35341e3a608c77660f0
Sep 11 04:15:17.058097 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:17.058059    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86" event={"ID":"d209c456-cee2-4962-a671-d7c56c855aec","Type":"ContainerStarted","Data":"78fdde8dc638df2442c201fc839ac2d1af71beab60a3efa1344ab1773f8ddcf2"}
Sep 11 04:15:17.058286 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:17.058124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86" event={"ID":"d209c456-cee2-4962-a671-d7c56c855aec","Type":"ContainerStarted","Data":"1d454e98d98181f39521e67767e0ae042561ef2b6190e35341e3a608c77660f0"}
Sep 11 04:15:17.058369 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:17.058348    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:17.079474 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:17.079397    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86" podStartSLOduration=1.079376696 podStartE2EDuration="1.079376696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:17.077828884 +0000 UTC m=+1514.943848621" watchObservedRunningTime="2026-09-11 04:15:17.079376696 +0000 UTC m=+1514.945396434"
Sep 11 04:15:22.376548 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:22.376515    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:15:28.066908 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:28.066831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:46.007546 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.007508    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"]
Sep 11 04:15:46.008166 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.007731    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86" podUID="d209c456-cee2-4962-a671-d7c56c855aec" containerName="payload-pre-processing" containerID="cri-o://78fdde8dc638df2442c201fc839ac2d1af71beab60a3efa1344ab1773f8ddcf2" gracePeriod=30
Sep 11 04:15:46.207172 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.207133    2566 generic.go:356] "Generic (PLEG): container finished" podID="d209c456-cee2-4962-a671-d7c56c855aec" containerID="78fdde8dc638df2442c201fc839ac2d1af71beab60a3efa1344ab1773f8ddcf2" exitCode=0
Sep 11 04:15:46.207356 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.207180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86" event={"ID":"d209c456-cee2-4962-a671-d7c56c855aec","Type":"ContainerDied","Data":"78fdde8dc638df2442c201fc839ac2d1af71beab60a3efa1344ab1773f8ddcf2"}
Sep 11 04:15:46.262949 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.262884    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:46.305441 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.305389    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p8sb\" (UniqueName: \"kubernetes.io/projected/d209c456-cee2-4962-a671-d7c56c855aec-kube-api-access-7p8sb\") pod \"d209c456-cee2-4962-a671-d7c56c855aec\" (UID: \"d209c456-cee2-4962-a671-d7c56c855aec\") "
Sep 11 04:15:46.305654 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.305513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d209c456-cee2-4962-a671-d7c56c855aec-plugins-config-volume\") pod \"d209c456-cee2-4962-a671-d7c56c855aec\" (UID: \"d209c456-cee2-4962-a671-d7c56c855aec\") "
Sep 11 04:15:46.306476 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.306434    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d209c456-cee2-4962-a671-d7c56c855aec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d209c456-cee2-4962-a671-d7c56c855aec" (UID: "d209c456-cee2-4962-a671-d7c56c855aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:15:46.313744 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.313708    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d209c456-cee2-4962-a671-d7c56c855aec-kube-api-access-7p8sb" (OuterVolumeSpecName: "kube-api-access-7p8sb") pod "d209c456-cee2-4962-a671-d7c56c855aec" (UID: "d209c456-cee2-4962-a671-d7c56c855aec"). InnerVolumeSpecName "kube-api-access-7p8sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:15:46.406870 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.406832    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d209c456-cee2-4962-a671-d7c56c855aec-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:15:46.406870 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:46.406862    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8sb\" (UniqueName: \"kubernetes.io/projected/d209c456-cee2-4962-a671-d7c56c855aec-kube-api-access-7p8sb\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:15:47.214092 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:47.214056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86" event={"ID":"d209c456-cee2-4962-a671-d7c56c855aec","Type":"ContainerDied","Data":"1d454e98d98181f39521e67767e0ae042561ef2b6190e35341e3a608c77660f0"}
Sep 11 04:15:47.214599 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:47.214122    2566 scope.go:117] "RemoveContainer" containerID="78fdde8dc638df2442c201fc839ac2d1af71beab60a3efa1344ab1773f8ddcf2"
Sep 11 04:15:47.214599 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:47.214071    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"
Sep 11 04:15:47.228983 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:47.228959    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"]
Sep 11 04:15:47.232575 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:47.232546    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-27c12d-57bfd4c754-s8f86"]
Sep 11 04:15:48.612763 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:48.612728    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d209c456-cee2-4962-a671-d7c56c855aec" path="/var/lib/kubelet/pods/d209c456-cee2-4962-a671-d7c56c855aec/volumes"
Sep 11 04:15:50.488565 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:15:50.488527    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:16:09.376967 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:09.376928    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:16:11.719548 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:11.719512    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"]
Sep 11 04:16:11.724253 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:11.724225    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ef4b62-29818335-b75fb"]
Sep 11 04:16:11.753286 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:11.753246    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"]
Sep 11 04:16:11.753553 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:11.753521    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj" podUID="584ecaf6-96e0-4937-ac48-c3008904cfe5" containerName="payload-pre-processing" containerID="cri-o://fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14" gracePeriod=30
Sep 11 04:16:12.023902 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.023875    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:16:12.062111 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.062069    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584ecaf6-96e0-4937-ac48-c3008904cfe5-plugins-config-volume\") pod \"584ecaf6-96e0-4937-ac48-c3008904cfe5\" (UID: \"584ecaf6-96e0-4937-ac48-c3008904cfe5\") "
Sep 11 04:16:12.062286 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.062122    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5hmg\" (UniqueName: \"kubernetes.io/projected/584ecaf6-96e0-4937-ac48-c3008904cfe5-kube-api-access-h5hmg\") pod \"584ecaf6-96e0-4937-ac48-c3008904cfe5\" (UID: \"584ecaf6-96e0-4937-ac48-c3008904cfe5\") "
Sep 11 04:16:12.062482 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.062456    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584ecaf6-96e0-4937-ac48-c3008904cfe5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "584ecaf6-96e0-4937-ac48-c3008904cfe5" (UID: "584ecaf6-96e0-4937-ac48-c3008904cfe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:16:12.062547 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.062535    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584ecaf6-96e0-4937-ac48-c3008904cfe5-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:16:12.064199 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.064176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584ecaf6-96e0-4937-ac48-c3008904cfe5-kube-api-access-h5hmg" (OuterVolumeSpecName: "kube-api-access-h5hmg") pod "584ecaf6-96e0-4937-ac48-c3008904cfe5" (UID: "584ecaf6-96e0-4937-ac48-c3008904cfe5"). InnerVolumeSpecName "kube-api-access-h5hmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:16:12.163007 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.162967    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5hmg\" (UniqueName: \"kubernetes.io/projected/584ecaf6-96e0-4937-ac48-c3008904cfe5-kube-api-access-h5hmg\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:16:12.340480 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.340355    2566 generic.go:356] "Generic (PLEG): container finished" podID="584ecaf6-96e0-4937-ac48-c3008904cfe5" containerID="fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14" exitCode=0
Sep 11 04:16:12.340480 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.340433    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"
Sep 11 04:16:12.340707 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.340434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj" event={"ID":"584ecaf6-96e0-4937-ac48-c3008904cfe5","Type":"ContainerDied","Data":"fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14"}
Sep 11 04:16:12.340707 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.340577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj" event={"ID":"584ecaf6-96e0-4937-ac48-c3008904cfe5","Type":"ContainerDied","Data":"169ac1859e1ba7064cd9e10ba8208d78e7feaa992457ba98d65fa5714daaf883"}
Sep 11 04:16:12.340707 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.340599    2566 scope.go:117] "RemoveContainer" containerID="fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14"
Sep 11 04:16:12.349563 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.349542    2566 scope.go:117] "RemoveContainer" containerID="fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14"
Sep 11 04:16:12.349853 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:16:12.349835    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14\": container with ID starting with fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14 not found: ID does not exist" containerID="fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14"
Sep 11 04:16:12.349914 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.349864    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14"} err="failed to get container status \"fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14\": rpc error: code = NotFound desc = could not find container \"fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14\": container with ID starting with fbdf8361c3cbd3dde06e2039d034bd13e52c566dbc09c3298ef556afe26f9a14 not found: ID does not exist"
Sep 11 04:16:12.364573 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.364537    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"]
Sep 11 04:16:12.369946 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.369921    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ef4b62-85d4584f74-w6krj"]
Sep 11 04:16:12.615586 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.615501    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584ecaf6-96e0-4937-ac48-c3008904cfe5" path="/var/lib/kubelet/pods/584ecaf6-96e0-4937-ac48-c3008904cfe5/volumes"
Sep 11 04:16:12.615824 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:12.615804    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7474158b-b28a-498e-9e58-a1ca58fd79ae" path="/var/lib/kubelet/pods/7474158b-b28a-498e-9e58-a1ca58fd79ae/volumes"
Sep 11 04:16:17.271263 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:17.271231    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"]
Sep 11 04:16:21.778651 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:21.778619    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:16:32.290553 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.290506    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8" podUID="32c032ae-a660-45fa-9876-032e7c320ff3" containerName="main" containerID="cri-o://a8de040f045f29986dc1a263ceb5b245a9fe6e7ce94c62cc1a87ab09053a3fac" gracePeriod=45
Sep 11 04:16:32.448629 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.448598    2566 generic.go:356] "Generic (PLEG): container finished" podID="32c032ae-a660-45fa-9876-032e7c320ff3" containerID="a8de040f045f29986dc1a263ceb5b245a9fe6e7ce94c62cc1a87ab09053a3fac" exitCode=0
Sep 11 04:16:32.448767 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.448663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8" event={"ID":"32c032ae-a660-45fa-9876-032e7c320ff3","Type":"ContainerDied","Data":"a8de040f045f29986dc1a263ceb5b245a9fe6e7ce94c62cc1a87ab09053a3fac"}
Sep 11 04:16:32.542054 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.542004    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:16:32.649653 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.649625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-dshm\") pod \"32c032ae-a660-45fa-9876-032e7c320ff3\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") "
Sep 11 04:16:32.649806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.649687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-model-cache\") pod \"32c032ae-a660-45fa-9876-032e7c320ff3\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") "
Sep 11 04:16:32.649806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.649711    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-home\") pod \"32c032ae-a660-45fa-9876-032e7c320ff3\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") "
Sep 11 04:16:32.649806 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.649802    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s86zp\" (UniqueName: \"kubernetes.io/projected/32c032ae-a660-45fa-9876-032e7c320ff3-kube-api-access-s86zp\") pod \"32c032ae-a660-45fa-9876-032e7c320ff3\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") "
Sep 11 04:16:32.649937 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.649819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c032ae-a660-45fa-9876-032e7c320ff3-tls-certs\") pod \"32c032ae-a660-45fa-9876-032e7c320ff3\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") "
Sep 11 04:16:32.649937 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.649839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-tmp-dir\") pod \"32c032ae-a660-45fa-9876-032e7c320ff3\" (UID: \"32c032ae-a660-45fa-9876-032e7c320ff3\") "
Sep 11 04:16:32.650044 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.649955    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-model-cache" (OuterVolumeSpecName: "model-cache") pod "32c032ae-a660-45fa-9876-032e7c320ff3" (UID: "32c032ae-a660-45fa-9876-032e7c320ff3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:32.650044 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.649997    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-home" (OuterVolumeSpecName: "home") pod "32c032ae-a660-45fa-9876-032e7c320ff3" (UID: "32c032ae-a660-45fa-9876-032e7c320ff3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:32.650204 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.650169    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-model-cache\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:16:32.650204 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.650196    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-home\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:16:32.650352 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.650213    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32c032ae-a660-45fa-9876-032e7c320ff3" (UID: "32c032ae-a660-45fa-9876-032e7c320ff3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:32.651779 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.651758    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-dshm" (OuterVolumeSpecName: "dshm") pod "32c032ae-a660-45fa-9876-032e7c320ff3" (UID: "32c032ae-a660-45fa-9876-032e7c320ff3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:16:32.651893 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.651877    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c032ae-a660-45fa-9876-032e7c320ff3-kube-api-access-s86zp" (OuterVolumeSpecName: "kube-api-access-s86zp") pod "32c032ae-a660-45fa-9876-032e7c320ff3" (UID: "32c032ae-a660-45fa-9876-032e7c320ff3"). InnerVolumeSpecName "kube-api-access-s86zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:16:32.651941 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.651913    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c032ae-a660-45fa-9876-032e7c320ff3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32c032ae-a660-45fa-9876-032e7c320ff3" (UID: "32c032ae-a660-45fa-9876-032e7c320ff3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:16:32.751689 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.751655    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s86zp\" (UniqueName: \"kubernetes.io/projected/32c032ae-a660-45fa-9876-032e7c320ff3-kube-api-access-s86zp\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:16:32.751689 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.751684    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c032ae-a660-45fa-9876-032e7c320ff3-tls-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:16:32.751689 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.751695    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-tmp-dir\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:16:32.751915 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:32.751702    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c032ae-a660-45fa-9876-032e7c320ff3-dshm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:16:33.455155 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:33.455123    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"
Sep 11 04:16:33.455585 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:33.455123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8" event={"ID":"32c032ae-a660-45fa-9876-032e7c320ff3","Type":"ContainerDied","Data":"04bc29539d06fd8b3e1daff3f0539ba6188bef76c01b603482d57432467a2a87"}
Sep 11 04:16:33.455585 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:33.455242    2566 scope.go:117] "RemoveContainer" containerID="a8de040f045f29986dc1a263ceb5b245a9fe6e7ce94c62cc1a87ab09053a3fac"
Sep 11 04:16:33.476372 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:33.476350    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"]
Sep 11 04:16:33.481559 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:33.481536    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ef4b62/ipp-route-ef4b62-kserve-548c545f87-7czn8"]
Sep 11 04:16:34.615781 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:16:34.615750    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c032ae-a660-45fa-9876-032e7c320ff3" path="/var/lib/kubelet/pods/32c032ae-a660-45fa-9876-032e7c320ff3/volumes"
Sep 11 04:17:16.791520 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:16.791483    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"]
Sep 11 04:17:16.792420 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:16.792350    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6" podUID="bd6826ca-b122-4a2e-b7d3-85c49af57968" containerName="payload-processing" containerID="cri-o://c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee" gracePeriod=30
Sep 11 04:17:17.064805 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.064771    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:17:17.151196 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.151155    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5csf\" (UniqueName: \"kubernetes.io/projected/bd6826ca-b122-4a2e-b7d3-85c49af57968-kube-api-access-g5csf\") pod \"bd6826ca-b122-4a2e-b7d3-85c49af57968\" (UID: \"bd6826ca-b122-4a2e-b7d3-85c49af57968\") "
Sep 11 04:17:17.151388 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.151258    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6826ca-b122-4a2e-b7d3-85c49af57968-plugins-config-volume\") pod \"bd6826ca-b122-4a2e-b7d3-85c49af57968\" (UID: \"bd6826ca-b122-4a2e-b7d3-85c49af57968\") "
Sep 11 04:17:17.151679 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.151656    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6826ca-b122-4a2e-b7d3-85c49af57968-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd6826ca-b122-4a2e-b7d3-85c49af57968" (UID: "bd6826ca-b122-4a2e-b7d3-85c49af57968"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:17:17.153538 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.153499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6826ca-b122-4a2e-b7d3-85c49af57968-kube-api-access-g5csf" (OuterVolumeSpecName: "kube-api-access-g5csf") pod "bd6826ca-b122-4a2e-b7d3-85c49af57968" (UID: "bd6826ca-b122-4a2e-b7d3-85c49af57968"). InnerVolumeSpecName "kube-api-access-g5csf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:17:17.252237 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.252185    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5csf\" (UniqueName: \"kubernetes.io/projected/bd6826ca-b122-4a2e-b7d3-85c49af57968-kube-api-access-g5csf\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:17.252237 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.252223    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6826ca-b122-4a2e-b7d3-85c49af57968-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:17.704483 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.704428    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd6826ca-b122-4a2e-b7d3-85c49af57968" containerID="c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee" exitCode=0
Sep 11 04:17:17.704677 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.704496    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"
Sep 11 04:17:17.704677 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.704497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6" event={"ID":"bd6826ca-b122-4a2e-b7d3-85c49af57968","Type":"ContainerDied","Data":"c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee"}
Sep 11 04:17:17.704677 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.704560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6" event={"ID":"bd6826ca-b122-4a2e-b7d3-85c49af57968","Type":"ContainerDied","Data":"fba0efab65acea385d067e05c6541c8f90af5f02bb1435cf29457b79573a721c"}
Sep 11 04:17:17.704677 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.704576    2566 scope.go:117] "RemoveContainer" containerID="c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee"
Sep 11 04:17:17.713893 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.713868    2566 scope.go:117] "RemoveContainer" containerID="c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee"
Sep 11 04:17:17.714233 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:17:17.714215    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee\": container with ID starting with c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee not found: ID does not exist" containerID="c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee"
Sep 11 04:17:17.714282 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.714242    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee"} err="failed to get container status \"c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee\": rpc error: code = NotFound desc = could not find container \"c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee\": container with ID starting with c1733c9c8a5847fbc02641ca07f2eea5f5c544f69c26b2b6beb4f8e1318057ee not found: ID does not exist"
Sep 11 04:17:17.727429 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.727380    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"]
Sep 11 04:17:17.731338 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:17.731303    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85dc29-59cb57985c-77vr6"]
Sep 11 04:17:18.614742 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:18.614702    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd6826ca-b122-4a2e-b7d3-85c49af57968" path="/var/lib/kubelet/pods/bd6826ca-b122-4a2e-b7d3-85c49af57968/volumes"
Sep 11 04:17:19.983735 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:19.983701    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:17:23.784435 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:23.784372    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"]
Sep 11 04:17:23.784892 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:23.784620    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy" containerID="cri-o://0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d" gracePeriod=30
Sep 11 04:17:29.051633 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.051546    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:17:29.169660 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169602    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-credential-socket\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.169660 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-certs\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.169943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169739    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-socket\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.169943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169756    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df030f2a-b43e-4155-ae88-fdbb43d924a8-istiod-ca-cert\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.169943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169779    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-token\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.169943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169800    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-envoy\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.169943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169824    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-data\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.169943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnmb5\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-kube-api-access-qnmb5\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.169943 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-podinfo\") pod \"df030f2a-b43e-4155-ae88-fdbb43d924a8\" (UID: \"df030f2a-b43e-4155-ae88-fdbb43d924a8\") "
Sep 11 04:17:29.170308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:17:29.170308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.170165    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:17:29.170308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.170176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:17:29.170308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.169981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:17:29.170308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.170271    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.170308 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.170300    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.170608 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.170315    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.170608 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.170328    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.170608 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.170464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df030f2a-b43e-4155-ae88-fdbb43d924a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:17:29.172157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.172125    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:17:29.172157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.172150    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:17:29.172478 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.172392    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-kube-api-access-qnmb5" (OuterVolumeSpecName: "kube-api-access-qnmb5") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "kube-api-access-qnmb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:17:29.172478 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.172393    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df030f2a-b43e-4155-ae88-fdbb43d924a8" (UID: "df030f2a-b43e-4155-ae88-fdbb43d924a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:17:29.271344 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.271301    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.271344 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.271337    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df030f2a-b43e-4155-ae88-fdbb43d924a8-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.271344 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.271347    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.271344 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.271355    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df030f2a-b43e-4155-ae88-fdbb43d924a8-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.271669 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.271364    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnmb5\" (UniqueName: \"kubernetes.io/projected/df030f2a-b43e-4155-ae88-fdbb43d924a8-kube-api-access-qnmb5\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:29.771580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.771544    2566 generic.go:356] "Generic (PLEG): container finished" podID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerID="0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d" exitCode=0
Sep 11 04:17:29.771764 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.771594    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"
Sep 11 04:17:29.771764 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.771641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" event={"ID":"df030f2a-b43e-4155-ae88-fdbb43d924a8","Type":"ContainerDied","Data":"0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d"}
Sep 11 04:17:29.771764 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.771698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k" event={"ID":"df030f2a-b43e-4155-ae88-fdbb43d924a8","Type":"ContainerDied","Data":"b2458c4aeb7ebbf66a22d74f1a8444b61b31d2c29943f3c1e284426aa19d8c4a"}
Sep 11 04:17:29.771764 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.771720    2566 scope.go:117] "RemoveContainer" containerID="0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d"
Sep 11 04:17:29.782261 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.782231    2566 scope.go:117] "RemoveContainer" containerID="14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f"
Sep 11 04:17:29.793662 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.793634    2566 scope.go:117] "RemoveContainer" containerID="0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d"
Sep 11 04:17:29.794098 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:17:29.794076    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d\": container with ID starting with 0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d not found: ID does not exist" containerID="0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d"
Sep 11 04:17:29.794166 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.794109    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d"} err="failed to get container status \"0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d\": rpc error: code = NotFound desc = could not find container \"0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d\": container with ID starting with 0dd4c015e8fd214fc54cc2866df40a48f03f699581e89eabb2bb16026cb1c70d not found: ID does not exist"
Sep 11 04:17:29.794166 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.794127    2566 scope.go:117] "RemoveContainer" containerID="14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f"
Sep 11 04:17:29.794427 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:17:29.794394    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f\": container with ID starting with 14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f not found: ID does not exist" containerID="14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f"
Sep 11 04:17:29.794471 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.794427    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f"} err="failed to get container status \"14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f\": rpc error: code = NotFound desc = could not find container \"14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f\": container with ID starting with 14e111882758d401bf73508bb69994009d0a136b7420d37664ca4a75d0d8154f not found: ID does not exist"
Sep 11 04:17:29.794722 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.794703    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"]
Sep 11 04:17:29.798525 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:29.798491    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-85dc29-openshift-default-58f665bb5d-4nt4k"]
Sep 11 04:17:30.616363 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:30.616266    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" path="/var/lib/kubelet/pods/df030f2a-b43e-4155-ae88-fdbb43d924a8/volumes"
Sep 11 04:17:37.385384 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:37.385337    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:17:52.523812 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.523773    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"]
Sep 11 04:17:52.524219 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.524042    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc" podUID="35cd3324-f1c8-489f-a93d-95695af0a931" containerName="payload-processing" containerID="cri-o://476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9" gracePeriod=30
Sep 11 04:17:52.783200 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.783145    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:17:52.887667 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.887635    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th7fm\" (UniqueName: \"kubernetes.io/projected/35cd3324-f1c8-489f-a93d-95695af0a931-kube-api-access-th7fm\") pod \"35cd3324-f1c8-489f-a93d-95695af0a931\" (UID: \"35cd3324-f1c8-489f-a93d-95695af0a931\") "
Sep 11 04:17:52.887847 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.887685    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cd3324-f1c8-489f-a93d-95695af0a931-plugins-config-volume\") pod \"35cd3324-f1c8-489f-a93d-95695af0a931\" (UID: \"35cd3324-f1c8-489f-a93d-95695af0a931\") "
Sep 11 04:17:52.888085 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.888040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cd3324-f1c8-489f-a93d-95695af0a931-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35cd3324-f1c8-489f-a93d-95695af0a931" (UID: "35cd3324-f1c8-489f-a93d-95695af0a931"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:17:52.890374 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.890342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cd3324-f1c8-489f-a93d-95695af0a931-kube-api-access-th7fm" (OuterVolumeSpecName: "kube-api-access-th7fm") pod "35cd3324-f1c8-489f-a93d-95695af0a931" (UID: "35cd3324-f1c8-489f-a93d-95695af0a931"). InnerVolumeSpecName "kube-api-access-th7fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:17:52.894770 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.894746    2566 generic.go:356] "Generic (PLEG): container finished" podID="35cd3324-f1c8-489f-a93d-95695af0a931" containerID="476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9" exitCode=0
Sep 11 04:17:52.894900 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.894802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc" event={"ID":"35cd3324-f1c8-489f-a93d-95695af0a931","Type":"ContainerDied","Data":"476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9"}
Sep 11 04:17:52.894900 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.894840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc" event={"ID":"35cd3324-f1c8-489f-a93d-95695af0a931","Type":"ContainerDied","Data":"366dadce7cdb66d619e94671e9878193f2d6525a7110635c0ccf208ca6f9f30d"}
Sep 11 04:17:52.894900 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.894861    2566 scope.go:117] "RemoveContainer" containerID="476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9"
Sep 11 04:17:52.894900 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.894803    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"
Sep 11 04:17:52.903109 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.903072    2566 scope.go:117] "RemoveContainer" containerID="476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9"
Sep 11 04:17:52.903393 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:17:52.903370    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9\": container with ID starting with 476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9 not found: ID does not exist" containerID="476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9"
Sep 11 04:17:52.903469 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.903419    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9"} err="failed to get container status \"476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9\": rpc error: code = NotFound desc = could not find container \"476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9\": container with ID starting with 476862acedce08ea6229cf96dc1c5104820f3b7f1ba5905fab9574a2ae868dd9 not found: ID does not exist"
Sep 11 04:17:52.916948 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.916922    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"]
Sep 11 04:17:52.921059 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.921040    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c158b2-5c74d58bdc-jn6vc"]
Sep 11 04:17:52.989157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.989137    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7fm\" (UniqueName: \"kubernetes.io/projected/35cd3324-f1c8-489f-a93d-95695af0a931-kube-api-access-th7fm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:52.989157 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:52.989157    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cd3324-f1c8-489f-a93d-95695af0a931-plugins-config-volume\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:17:54.612564 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:54.612534    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35cd3324-f1c8-489f-a93d-95695af0a931" path="/var/lib/kubelet/pods/35cd3324-f1c8-489f-a93d-95695af0a931/volumes"
Sep 11 04:17:55.095965 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:55.095934    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:17:59.601647 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:59.601612    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"]
Sep 11 04:17:59.602055 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:17:59.601854    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy" containerID="cri-o://c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340" gracePeriod=30
Sep 11 04:18:02.377321 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.377286    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-lnzhx"]
Sep 11 04:18:02.377793 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.377563    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-848cd9bd7-lnzhx" podUID="bb8c3ca4-e38e-420e-8bc0-00d008dad517" containerName="manager" containerID="cri-o://ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28" gracePeriod=30
Sep 11 04:18:02.626558 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.626536    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 04:18:02.665472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.665389    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv8hb\" (UniqueName: \"kubernetes.io/projected/bb8c3ca4-e38e-420e-8bc0-00d008dad517-kube-api-access-wv8hb\") pod \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") "
Sep 11 04:18:02.665472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.665446    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-webhook-cert\") pod \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") "
Sep 11 04:18:02.665693 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.665479    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs\") pod \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\" (UID: \"bb8c3ca4-e38e-420e-8bc0-00d008dad517\") "
Sep 11 04:18:02.667578 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.667539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb8c3ca4-e38e-420e-8bc0-00d008dad517" (UID: "bb8c3ca4-e38e-420e-8bc0-00d008dad517"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:18:02.667963 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.667935    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bb8c3ca4-e38e-420e-8bc0-00d008dad517" (UID: "bb8c3ca4-e38e-420e-8bc0-00d008dad517"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:18:02.668036 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.667985    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8c3ca4-e38e-420e-8bc0-00d008dad517-kube-api-access-wv8hb" (OuterVolumeSpecName: "kube-api-access-wv8hb") pod "bb8c3ca4-e38e-420e-8bc0-00d008dad517" (UID: "bb8c3ca4-e38e-420e-8bc0-00d008dad517"). InnerVolumeSpecName "kube-api-access-wv8hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:18:02.766527 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.766498    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv8hb\" (UniqueName: \"kubernetes.io/projected/bb8c3ca4-e38e-420e-8bc0-00d008dad517-kube-api-access-wv8hb\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:02.766527 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.766525    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-webhook-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:02.766714 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.766537    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c3ca4-e38e-420e-8bc0-00d008dad517-metrics-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:02.942760 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.942680    2566 generic.go:356] "Generic (PLEG): container finished" podID="bb8c3ca4-e38e-420e-8bc0-00d008dad517" containerID="ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28" exitCode=0
Sep 11 04:18:02.942760 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.942743    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-lnzhx"
Sep 11 04:18:02.942931 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.942769    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-lnzhx" event={"ID":"bb8c3ca4-e38e-420e-8bc0-00d008dad517","Type":"ContainerDied","Data":"ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28"}
Sep 11 04:18:02.942931 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.942846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-lnzhx" event={"ID":"bb8c3ca4-e38e-420e-8bc0-00d008dad517","Type":"ContainerDied","Data":"5aaaf73b5a8c5755fe22bb27467c249621953e8d6ce0b86f29e1e4872b2a572b"}
Sep 11 04:18:02.942931 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.942870    2566 scope.go:117] "RemoveContainer" containerID="ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28"
Sep 11 04:18:02.951786 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.951771    2566 scope.go:117] "RemoveContainer" containerID="ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28"
Sep 11 04:18:02.952027 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:18:02.952009    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28\": container with ID starting with ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28 not found: ID does not exist" containerID="ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28"
Sep 11 04:18:02.952079 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.952034    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28"} err="failed to get container status \"ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28\": rpc error: code = NotFound desc = could not find container \"ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28\": container with ID starting with ce2bea4dead38a3c42a7f7cc6c1ae360d0c23c87f687d60d2d07c87e168d3b28 not found: ID does not exist"
Sep 11 04:18:02.963013 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.962993    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-lnzhx"]
Sep 11 04:18:02.967171 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:02.967152    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-lnzhx"]
Sep 11 04:18:03.684692 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:03.684659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:18:04.612517 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.612475    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb8c3ca4-e38e-420e-8bc0-00d008dad517" path="/var/lib/kubelet/pods/bb8c3ca4-e38e-420e-8bc0-00d008dad517/volumes"
Sep 11 04:18:04.858347 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.858317    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:18:04.955965 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.955870    2566 generic.go:356] "Generic (PLEG): container finished" podID="66920c5d-f81b-444d-806c-564051d7de1e" containerID="c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340" exitCode=0
Sep 11 04:18:04.955965 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.955954    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"
Sep 11 04:18:04.956327 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.955956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" event={"ID":"66920c5d-f81b-444d-806c-564051d7de1e","Type":"ContainerDied","Data":"c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340"}
Sep 11 04:18:04.956327 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.956041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q" event={"ID":"66920c5d-f81b-444d-806c-564051d7de1e","Type":"ContainerDied","Data":"64324143885eab2aecbfa004ecdb776d590b4c7d8036377ff048d91eb6636751"}
Sep 11 04:18:04.956327 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.956076    2566 scope.go:117] "RemoveContainer" containerID="c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340"
Sep 11 04:18:04.966542 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.966451    2566 scope.go:117] "RemoveContainer" containerID="6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f"
Sep 11 04:18:04.975564 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.975544    2566 scope.go:117] "RemoveContainer" containerID="c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340"
Sep 11 04:18:04.975868 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:18:04.975833    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340\": container with ID starting with c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340 not found: ID does not exist" containerID="c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340"
Sep 11 04:18:04.975958 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.975879    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340"} err="failed to get container status \"c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340\": rpc error: code = NotFound desc = could not find container \"c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340\": container with ID starting with c0d8fac32f16dc0c549bb0475e457e409689de77f4c357004ebc9ccc4df34340 not found: ID does not exist"
Sep 11 04:18:04.975958 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.975904    2566 scope.go:117] "RemoveContainer" containerID="6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f"
Sep 11 04:18:04.976225 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:18:04.976204    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f\": container with ID starting with 6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f not found: ID does not exist" containerID="6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f"
Sep 11 04:18:04.976300 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.976231    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f"} err="failed to get container status \"6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f\": rpc error: code = NotFound desc = could not find container \"6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f\": container with ID starting with 6874d197b8ac1c9a820a97f64646266202402bbea52e5875597ccb86102bd97f not found: ID does not exist"
Sep 11 04:18:04.983518 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-envoy\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.983624 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983528    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-certs\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.983624 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983547    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-credential-socket\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.983624 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983602    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66920c5d-f81b-444d-806c-564051d7de1e-istio-podinfo\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.983819 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-istio-token\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.983819 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983665    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-socket\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.983819 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66920c5d-f81b-444d-806c-564051d7de1e-istiod-ca-cert\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.983819 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983764    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-data\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.983819 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983790    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxhzf\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-kube-api-access-hxhzf\") pod \"66920c5d-f81b-444d-806c-564051d7de1e\" (UID: \"66920c5d-f81b-444d-806c-564051d7de1e\") "
Sep 11 04:18:04.984170 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983818    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:04.984170 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.983899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:04.984170 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.984086    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:04.984170 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.984105    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-credential-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:04.984170 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.984159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66920c5d-f81b-444d-806c-564051d7de1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:18:04.984563 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.984222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:04.984917 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.984874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:04.986591 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.986563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:18:04.986591 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.986574    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66920c5d-f81b-444d-806c-564051d7de1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 04:18:04.986750 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.986573    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:18:04.987272 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:04.987240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-kube-api-access-hxhzf" (OuterVolumeSpecName: "kube-api-access-hxhzf") pod "66920c5d-f81b-444d-806c-564051d7de1e" (UID: "66920c5d-f81b-444d-806c-564051d7de1e"). InnerVolumeSpecName "kube-api-access-hxhzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:18:05.084938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.084913    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-envoy\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:05.084938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.084934    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66920c5d-f81b-444d-806c-564051d7de1e-istio-podinfo\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:05.084938 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.084944    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-istio-token\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:05.085139 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.084953    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-workload-socket\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:05.085139 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.084962    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66920c5d-f81b-444d-806c-564051d7de1e-istiod-ca-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:05.085139 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.084970    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66920c5d-f81b-444d-806c-564051d7de1e-istio-data\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:05.085139 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.084978    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhzf\" (UniqueName: \"kubernetes.io/projected/66920c5d-f81b-444d-806c-564051d7de1e-kube-api-access-hxhzf\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:18:05.277972 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.277945    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"]
Sep 11 04:18:05.282675 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:05.282655    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c158b2-openshift-default-95b6c476d-pxb5q"]
Sep 11 04:18:06.611803 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:18:06.611768    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66920c5d-f81b-444d-806c-564051d7de1e" path="/var/lib/kubelet/pods/66920c5d-f81b-444d-806c-564051d7de1e/volumes"
Sep 11 04:19:03.810944 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.810911    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-848cd9bd7-wwt4m"]
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811217    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811228    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811236    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8c3ca4-e38e-420e-8bc0-00d008dad517" containerName="manager"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811242    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8c3ca4-e38e-420e-8bc0-00d008dad517" containerName="manager"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811250    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c032ae-a660-45fa-9876-032e7c320ff3" containerName="main"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811255    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c032ae-a660-45fa-9876-032e7c320ff3" containerName="main"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811265    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811271    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811276    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811280    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811287    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811291    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811297    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d209c456-cee2-4962-a671-d7c56c855aec" containerName="payload-pre-processing"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811302    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d209c456-cee2-4962-a671-d7c56c855aec" containerName="payload-pre-processing"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811312    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584ecaf6-96e0-4937-ac48-c3008904cfe5" containerName="payload-pre-processing"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811317    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="584ecaf6-96e0-4937-ac48-c3008904cfe5" containerName="payload-pre-processing"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811324    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cd3324-f1c8-489f-a93d-95695af0a931" containerName="payload-processing"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811329    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cd3324-f1c8-489f-a93d-95695af0a931" containerName="payload-processing"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811334    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd6826ca-b122-4a2e-b7d3-85c49af57968" containerName="payload-processing"
Sep 11 04:19:03.811332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811339    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd6826ca-b122-4a2e-b7d3-85c49af57968" containerName="payload-processing"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811390    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811397    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="584ecaf6-96e0-4937-ac48-c3008904cfe5" containerName="payload-pre-processing"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811416    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d209c456-cee2-4962-a671-d7c56c855aec" containerName="payload-pre-processing"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811423    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c032ae-a660-45fa-9876-032e7c320ff3" containerName="main"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811430    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb8c3ca4-e38e-420e-8bc0-00d008dad517" containerName="manager"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811435    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811441    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd6826ca-b122-4a2e-b7d3-85c49af57968" containerName="payload-processing"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811447    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df030f2a-b43e-4155-ae88-fdbb43d924a8" containerName="istio-proxy"
Sep 11 04:19:03.812021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.811453    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cd3324-f1c8-489f-a93d-95695af0a931" containerName="payload-processing"
Sep 11 04:19:03.815549 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.815533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.818817 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.818793    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 04:19:03.819002 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.818986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gp885\""
Sep 11 04:19:03.819059 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.819020    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 04:19:03.819106 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.819086    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 04:19:03.825548 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.825525    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-wwt4m"]
Sep 11 04:19:03.826613 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.826592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-metrics-certs\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.826721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.826637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwcb\" (UniqueName: \"kubernetes.io/projected/90113672-d74b-49d8-a2ea-383ec1198e1b-kube-api-access-4jwcb\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.826721 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.826662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-webhook-cert\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.927156 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.927130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-metrics-certs\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.927282 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.927180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwcb\" (UniqueName: \"kubernetes.io/projected/90113672-d74b-49d8-a2ea-383ec1198e1b-kube-api-access-4jwcb\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.927282 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.927209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-webhook-cert\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.929593 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.929568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-metrics-certs\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.929684 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.929616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-webhook-cert\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:03.935087 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:03.935064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwcb\" (UniqueName: \"kubernetes.io/projected/90113672-d74b-49d8-a2ea-383ec1198e1b-kube-api-access-4jwcb\") pod \"maas-controller-848cd9bd7-wwt4m\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") " pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:04.126161 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:04.126096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:04.411811 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:04.411787    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-wwt4m"]
Sep 11 04:19:04.414593 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:19:04.414561    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90113672_d74b_49d8_a2ea_383ec1198e1b.slice/crio-a61e6e436260846d8ff8f4f793a24603c1826f09dd2cc60bb17c95b6be675c95 WatchSource:0}: Error finding container a61e6e436260846d8ff8f4f793a24603c1826f09dd2cc60bb17c95b6be675c95: Status 404 returned error can't find the container with id a61e6e436260846d8ff8f4f793a24603c1826f09dd2cc60bb17c95b6be675c95
Sep 11 04:19:05.233725 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:05.233693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-wwt4m" event={"ID":"90113672-d74b-49d8-a2ea-383ec1198e1b","Type":"ContainerStarted","Data":"45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e"}
Sep 11 04:19:05.234122 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:05.233742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-wwt4m" event={"ID":"90113672-d74b-49d8-a2ea-383ec1198e1b","Type":"ContainerStarted","Data":"a61e6e436260846d8ff8f4f793a24603c1826f09dd2cc60bb17c95b6be675c95"}
Sep 11 04:19:05.234122 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:05.233817    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:05.252948 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:05.252901    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-848cd9bd7-wwt4m" podStartSLOduration=1.876851896 podStartE2EDuration="2.252888772s" totalImagesPullingTime="376.036876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:19:05.252548746 +0000 UTC m=+1743.118568482" watchObservedRunningTime="2026-09-11 04:19:05.252888772 +0000 UTC m=+1743.118908506"
Sep 11 04:19:16.240697 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:16.240668    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:23.454522 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.454487    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-wwt4m"]
Sep 11 04:19:23.454956 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.454689    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-848cd9bd7-wwt4m" podUID="90113672-d74b-49d8-a2ea-383ec1198e1b" containerName="manager" containerID="cri-o://45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e" gracePeriod=30
Sep 11 04:19:23.692479 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.692458    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:23.782874 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.782847    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-metrics-certs\") pod \"90113672-d74b-49d8-a2ea-383ec1198e1b\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") "
Sep 11 04:19:23.783051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.782929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-webhook-cert\") pod \"90113672-d74b-49d8-a2ea-383ec1198e1b\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") "
Sep 11 04:19:23.783051 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.782999    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jwcb\" (UniqueName: \"kubernetes.io/projected/90113672-d74b-49d8-a2ea-383ec1198e1b-kube-api-access-4jwcb\") pod \"90113672-d74b-49d8-a2ea-383ec1198e1b\" (UID: \"90113672-d74b-49d8-a2ea-383ec1198e1b\") "
Sep 11 04:19:23.784853 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.784822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90113672-d74b-49d8-a2ea-383ec1198e1b" (UID: "90113672-d74b-49d8-a2ea-383ec1198e1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:19:23.784976 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.784907    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "90113672-d74b-49d8-a2ea-383ec1198e1b" (UID: "90113672-d74b-49d8-a2ea-383ec1198e1b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:19:23.784976 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.784957    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90113672-d74b-49d8-a2ea-383ec1198e1b-kube-api-access-4jwcb" (OuterVolumeSpecName: "kube-api-access-4jwcb") pod "90113672-d74b-49d8-a2ea-383ec1198e1b" (UID: "90113672-d74b-49d8-a2ea-383ec1198e1b"). InnerVolumeSpecName "kube-api-access-4jwcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:19:23.884400 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.884379    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-metrics-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:19:23.884509 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.884402    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90113672-d74b-49d8-a2ea-383ec1198e1b-webhook-cert\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:19:23.884509 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.884427    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwcb\" (UniqueName: \"kubernetes.io/projected/90113672-d74b-49d8-a2ea-383ec1198e1b-kube-api-access-4jwcb\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:19:23.990261 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:23.990236    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:19:24.328093 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.328066    2566 generic.go:356] "Generic (PLEG): container finished" podID="90113672-d74b-49d8-a2ea-383ec1198e1b" containerID="45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e" exitCode=0
Sep 11 04:19:24.328235 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.328116    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-wwt4m"
Sep 11 04:19:24.328235 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.328132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-wwt4m" event={"ID":"90113672-d74b-49d8-a2ea-383ec1198e1b","Type":"ContainerDied","Data":"45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e"}
Sep 11 04:19:24.328235 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.328171    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-wwt4m" event={"ID":"90113672-d74b-49d8-a2ea-383ec1198e1b","Type":"ContainerDied","Data":"a61e6e436260846d8ff8f4f793a24603c1826f09dd2cc60bb17c95b6be675c95"}
Sep 11 04:19:24.328235 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.328193    2566 scope.go:117] "RemoveContainer" containerID="45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e"
Sep 11 04:19:24.336484 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.336470    2566 scope.go:117] "RemoveContainer" containerID="45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e"
Sep 11 04:19:24.336737 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:19:24.336718    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e\": container with ID starting with 45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e not found: ID does not exist" containerID="45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e"
Sep 11 04:19:24.336807 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.336748    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e"} err="failed to get container status \"45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e\": rpc error: code = NotFound desc = could not find container \"45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e\": container with ID starting with 45a7f46f26af13638b07d5a9bc2b038e77a008571330850f7001d0631a8a129e not found: ID does not exist"
Sep 11 04:19:24.348477 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.348453    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-wwt4m"]
Sep 11 04:19:24.355590 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.355571    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-wwt4m"]
Sep 11 04:19:24.614478 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:24.614347    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90113672-d74b-49d8-a2ea-383ec1198e1b" path="/var/lib/kubelet/pods/90113672-d74b-49d8-a2ea-383ec1198e1b/volumes"
Sep 11 04:19:25.297880 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.297847    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-848cd9bd7-ccx4s"]
Sep 11 04:19:25.298175 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.298162    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90113672-d74b-49d8-a2ea-383ec1198e1b" containerName="manager"
Sep 11 04:19:25.298223 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.298176    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90113672-d74b-49d8-a2ea-383ec1198e1b" containerName="manager"
Sep 11 04:19:25.298258 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.298232    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90113672-d74b-49d8-a2ea-383ec1198e1b" containerName="manager"
Sep 11 04:19:25.298258 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.298241    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="66920c5d-f81b-444d-806c-564051d7de1e" containerName="istio-proxy"
Sep 11 04:19:25.302230 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.302214    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.305728 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.305700    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 04:19:25.305878 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.305746    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gp885\""
Sep 11 04:19:25.305878 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.305766    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 04:19:25.305878 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.305766    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 04:19:25.310427 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.310392    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-ccx4s"]
Sep 11 04:19:25.397146 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.397119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd990481-d6f9-4876-8a77-57f4fa010265-metrics-certs\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.397268 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.397151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9jz5\" (UniqueName: \"kubernetes.io/projected/cd990481-d6f9-4876-8a77-57f4fa010265-kube-api-access-f9jz5\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.397268 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.397188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd990481-d6f9-4876-8a77-57f4fa010265-webhook-cert\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.497655 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.497629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9jz5\" (UniqueName: \"kubernetes.io/projected/cd990481-d6f9-4876-8a77-57f4fa010265-kube-api-access-f9jz5\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.497750 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.497673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd990481-d6f9-4876-8a77-57f4fa010265-webhook-cert\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.497750 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.497714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd990481-d6f9-4876-8a77-57f4fa010265-metrics-certs\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.500093 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.500066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd990481-d6f9-4876-8a77-57f4fa010265-metrics-certs\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.500184 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.500125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd990481-d6f9-4876-8a77-57f4fa010265-webhook-cert\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.505462 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.505440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9jz5\" (UniqueName: \"kubernetes.io/projected/cd990481-d6f9-4876-8a77-57f4fa010265-kube-api-access-f9jz5\") pod \"maas-controller-848cd9bd7-ccx4s\" (UID: \"cd990481-d6f9-4876-8a77-57f4fa010265\") " pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.612756 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.612702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:25.880489 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:25.880465    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-848cd9bd7-ccx4s"]
Sep 11 04:19:25.882295 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:19:25.882268    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd990481_d6f9_4876_8a77_57f4fa010265.slice/crio-992df0b644f2185b0075bb0f05b0548baf8b3c4ba6c238cdbd50b40bfbf88a30 WatchSource:0}: Error finding container 992df0b644f2185b0075bb0f05b0548baf8b3c4ba6c238cdbd50b40bfbf88a30: Status 404 returned error can't find the container with id 992df0b644f2185b0075bb0f05b0548baf8b3c4ba6c238cdbd50b40bfbf88a30
Sep 11 04:19:26.341995 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:26.341972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-ccx4s" event={"ID":"cd990481-d6f9-4876-8a77-57f4fa010265","Type":"ContainerStarted","Data":"992df0b644f2185b0075bb0f05b0548baf8b3c4ba6c238cdbd50b40bfbf88a30"}
Sep 11 04:19:27.347472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:27.347434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-848cd9bd7-ccx4s" event={"ID":"cd990481-d6f9-4876-8a77-57f4fa010265","Type":"ContainerStarted","Data":"e9e74f63f3b406bec5d10305214728d1c951d3832f377bb67cc81a0138a65d10"}
Sep 11 04:19:27.347950 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:27.347530    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:27.366668 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:27.366625    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-848cd9bd7-ccx4s" podStartSLOduration=1.921962169 podStartE2EDuration="2.366611002s" totalImagesPullingTime="444.648833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:19:27.364254234 +0000 UTC m=+1765.230273968" watchObservedRunningTime="2026-09-11 04:19:27.366611002 +0000 UTC m=+1765.232630736"
Sep 11 04:19:38.353684 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:38.353658    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-848cd9bd7-ccx4s"
Sep 11 04:19:48.089166 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:48.089134    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:19:55.880991 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:19:55.880910    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:20:02.721144 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:20:02.721113    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:20:02.721927 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:20:02.721904    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vkpvh_d9d726a9-7a1b-4b79-8eb3-b41c7c1f817a/ovn-acl-logging/0.log"
Sep 11 04:20:04.076952 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:20:04.076921    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:20:21.583523 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:20:21.583486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:21:03.081151 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:21:03.081127    2566 scope.go:117] "RemoveContainer" containerID="3ef74651b0ceaa530d6e70dd8bc5fddda63842823b80ee51e0948f01bbcbf6e5"
Sep 11 04:21:49.990344 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:21:49.990307    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:21:57.782701 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:21:57.782655    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:22:05.875134 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:05.875100    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:22:11.904228 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.904196    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"]
Sep 11 04:22:11.907763 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.907742    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:11.910202 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.910172    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5ea2fe22-kserve-self-signed-certs\""
Sep 11 04:22:11.917219 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.917197    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"]
Sep 11 04:22:11.990225 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.990197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-tmp-dir\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:11.990351 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.990236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9f169a-9d45-428c-8753-3c9a221fa485-tls-certs\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:11.990351 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.990268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9f8v\" (UniqueName: \"kubernetes.io/projected/2b9f169a-9d45-428c-8753-3c9a221fa485-kube-api-access-q9f8v\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:11.990459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.990355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-dshm\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:11.990459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.990401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-home\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:11.990459 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:11.990457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-model-cache\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.091203 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.091169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-model-cache\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.091493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.091220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-tmp-dir\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.091493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.091260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9f169a-9d45-428c-8753-3c9a221fa485-tls-certs\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.091493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.091304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9f8v\" (UniqueName: \"kubernetes.io/projected/2b9f169a-9d45-428c-8753-3c9a221fa485-kube-api-access-q9f8v\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.091493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.091369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-dshm\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.091493 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.091398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-home\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.091801 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.091600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-model-cache\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.091966 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.091943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-tmp-dir\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.092043 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.092014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-home\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.093726 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.093705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-dshm\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.093840 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.093816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9f169a-9d45-428c-8753-3c9a221fa485-tls-certs\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.099579 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.099554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9f8v\" (UniqueName: \"kubernetes.io/projected/2b9f169a-9d45-428c-8753-3c9a221fa485-kube-api-access-q9f8v\") pod \"e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") " pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.216674 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.216653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:12.493641 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:12.493616    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"]
Sep 11 04:22:12.495036 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:22:12.495010    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b9f169a_9d45_428c_8753_3c9a221fa485.slice/crio-76831b9dbd1af7ca3db0d3188570d0255dcf400fb4bce6a71add830ae878f50f WatchSource:0}: Error finding container 76831b9dbd1af7ca3db0d3188570d0255dcf400fb4bce6a71add830ae878f50f: Status 404 returned error can't find the container with id 76831b9dbd1af7ca3db0d3188570d0255dcf400fb4bce6a71add830ae878f50f
Sep 11 04:22:13.085957 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:13.085927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh" event={"ID":"2b9f169a-9d45-428c-8753-3c9a221fa485","Type":"ContainerStarted","Data":"6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a"}
Sep 11 04:22:13.086330 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:13.085966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh" event={"ID":"2b9f169a-9d45-428c-8753-3c9a221fa485","Type":"ContainerStarted","Data":"76831b9dbd1af7ca3db0d3188570d0255dcf400fb4bce6a71add830ae878f50f"}
Sep 11 04:22:13.108652 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:13.108606    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh" podStartSLOduration=2.108591002 podStartE2EDuration="2.108591002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:22:13.106367079 +0000 UTC m=+1930.972386816" watchObservedRunningTime="2026-09-11 04:22:13.108591002 +0000 UTC m=+1930.974610738"
Sep 11 04:22:14.483389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:14.483357    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:22:22.217359 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:22.217318    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:22.217762 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:22.217377    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:22.229380 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:22.229359    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:23.139567 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:23.139539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:22:24.982389 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:24.982355    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:22:44.887332 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:44.887298    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"]
Sep 11 04:22:46.984960 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:46.984926    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:22:55.672478 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:55.672388    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:22:59.906580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:22:59.906519    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh" podUID="2b9f169a-9d45-428c-8753-3c9a221fa485" containerName="main" containerID="cri-o://6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a" gracePeriod=45
Sep 11 04:23:00.155974 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.155951    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:23:00.274622 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.274598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9f8v\" (UniqueName: \"kubernetes.io/projected/2b9f169a-9d45-428c-8753-3c9a221fa485-kube-api-access-q9f8v\") pod \"2b9f169a-9d45-428c-8753-3c9a221fa485\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") "
Sep 11 04:23:00.274765 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.274636    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-dshm\") pod \"2b9f169a-9d45-428c-8753-3c9a221fa485\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") "
Sep 11 04:23:00.274765 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.274657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-home\") pod \"2b9f169a-9d45-428c-8753-3c9a221fa485\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") "
Sep 11 04:23:00.274765 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.274675    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-tmp-dir\") pod \"2b9f169a-9d45-428c-8753-3c9a221fa485\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") "
Sep 11 04:23:00.274765 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.274714    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9f169a-9d45-428c-8753-3c9a221fa485-tls-certs\") pod \"2b9f169a-9d45-428c-8753-3c9a221fa485\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") "
Sep 11 04:23:00.274765 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.274748    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-model-cache\") pod \"2b9f169a-9d45-428c-8753-3c9a221fa485\" (UID: \"2b9f169a-9d45-428c-8753-3c9a221fa485\") "
Sep 11 04:23:00.275035 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.274906    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-home" (OuterVolumeSpecName: "home") pod "2b9f169a-9d45-428c-8753-3c9a221fa485" (UID: "2b9f169a-9d45-428c-8753-3c9a221fa485"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:23:00.275035 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.274977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b9f169a-9d45-428c-8753-3c9a221fa485" (UID: "2b9f169a-9d45-428c-8753-3c9a221fa485"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:23:00.275131 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.275066    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b9f169a-9d45-428c-8753-3c9a221fa485" (UID: "2b9f169a-9d45-428c-8753-3c9a221fa485"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:23:00.276831 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.276803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9f169a-9d45-428c-8753-3c9a221fa485-kube-api-access-q9f8v" (OuterVolumeSpecName: "kube-api-access-q9f8v") pod "2b9f169a-9d45-428c-8753-3c9a221fa485" (UID: "2b9f169a-9d45-428c-8753-3c9a221fa485"). InnerVolumeSpecName "kube-api-access-q9f8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:23:00.276831 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.276826    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9f169a-9d45-428c-8753-3c9a221fa485-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b9f169a-9d45-428c-8753-3c9a221fa485" (UID: "2b9f169a-9d45-428c-8753-3c9a221fa485"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:23:00.276975 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.276870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-dshm" (OuterVolumeSpecName: "dshm") pod "2b9f169a-9d45-428c-8753-3c9a221fa485" (UID: "2b9f169a-9d45-428c-8753-3c9a221fa485"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 04:23:00.303358 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.303328    2566 generic.go:356] "Generic (PLEG): container finished" podID="2b9f169a-9d45-428c-8753-3c9a221fa485" containerID="6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a" exitCode=0
Sep 11 04:23:00.303482 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.303362    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"
Sep 11 04:23:00.303482 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.303425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh" event={"ID":"2b9f169a-9d45-428c-8753-3c9a221fa485","Type":"ContainerDied","Data":"6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a"}
Sep 11 04:23:00.303482 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.303462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh" event={"ID":"2b9f169a-9d45-428c-8753-3c9a221fa485","Type":"ContainerDied","Data":"76831b9dbd1af7ca3db0d3188570d0255dcf400fb4bce6a71add830ae878f50f"}
Sep 11 04:23:00.303482 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.303480    2566 scope.go:117] "RemoveContainer" containerID="6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a"
Sep 11 04:23:00.311201 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.311183    2566 scope.go:117] "RemoveContainer" containerID="6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a"
Sep 11 04:23:00.311462 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:23:00.311441    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a\": container with ID starting with 6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a not found: ID does not exist" containerID="6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a"
Sep 11 04:23:00.311525 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.311471    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a"} err="failed to get container status \"6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a\": rpc error: code = NotFound desc = could not find container \"6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a\": container with ID starting with 6898306ff5a96b58b11374d2f3e89e752e0709e6f334ace68528ad575450753a not found: ID does not exist"
Sep 11 04:23:00.323102 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.323083    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"]
Sep 11 04:23:00.327466 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.327446    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5ea2fe22-kserve-6596fc5c75-jpbwh"]
Sep 11 04:23:00.375796 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.375778    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-model-cache\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:00.375796 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.375797    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9f8v\" (UniqueName: \"kubernetes.io/projected/2b9f169a-9d45-428c-8753-3c9a221fa485-kube-api-access-q9f8v\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:00.375919 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.375805    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-dshm\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:00.375919 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.375813    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-home\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:00.375919 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.375820    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9f169a-9d45-428c-8753-3c9a221fa485-tmp-dir\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:00.375919 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.375834    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9f169a-9d45-428c-8753-3c9a221fa485-tls-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:00.611910 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:00.611848    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b9f169a-9d45-428c-8753-3c9a221fa485" path="/var/lib/kubelet/pods/2b9f169a-9d45-428c-8753-3c9a221fa485/volumes"
Sep 11 04:23:03.079670 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:03.079629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:23:09.687259 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:09.687229    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:23:17.771555 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:17.771519    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:23:18.037191 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.037120    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"]
Sep 11 04:23:18.037472 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.037459    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9f169a-9d45-428c-8753-3c9a221fa485" containerName="main"
Sep 11 04:23:18.037523 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.037473    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9f169a-9d45-428c-8753-3c9a221fa485" containerName="main"
Sep 11 04:23:18.037557 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.037543    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9f169a-9d45-428c-8753-3c9a221fa485" containerName="main"
Sep 11 04:23:18.040550 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.040530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.055089 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.055062    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"]
Sep 11 04:23:18.125291 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.125266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6822a8d4-3241-4906-bff8-59f14d8e8818-metrics-certs\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.125440 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.125314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822a8d4-3241-4906-bff8-59f14d8e8818-maas-api-tls\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.125440 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.125421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnwj\" (UniqueName: \"kubernetes.io/projected/6822a8d4-3241-4906-bff8-59f14d8e8818-kube-api-access-hdnwj\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.125534 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.125468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6822a8d4-3241-4906-bff8-59f14d8e8818-openshift-service-ca\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.226179 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.226152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnwj\" (UniqueName: \"kubernetes.io/projected/6822a8d4-3241-4906-bff8-59f14d8e8818-kube-api-access-hdnwj\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.226307 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.226186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6822a8d4-3241-4906-bff8-59f14d8e8818-openshift-service-ca\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.226307 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.226213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6822a8d4-3241-4906-bff8-59f14d8e8818-metrics-certs\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.226450 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.226331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822a8d4-3241-4906-bff8-59f14d8e8818-maas-api-tls\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.226863 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.226839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6822a8d4-3241-4906-bff8-59f14d8e8818-openshift-service-ca\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.228699 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.228675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6822a8d4-3241-4906-bff8-59f14d8e8818-metrics-certs\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.228797 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.228727    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822a8d4-3241-4906-bff8-59f14d8e8818-maas-api-tls\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.234093 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.234070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnwj\" (UniqueName: \"kubernetes.io/projected/6822a8d4-3241-4906-bff8-59f14d8e8818-kube-api-access-hdnwj\") pod \"maas-api-6c9d99c6fc-drlr6\" (UID: \"6822a8d4-3241-4906-bff8-59f14d8e8818\") " pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.350156 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.350086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:18.621540 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.621515    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"]
Sep 11 04:23:18.624667 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:23:18.624626    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6822a8d4_3241_4906_bff8_59f14d8e8818.slice/crio-76b435e792cf1c845b3d227906f57bd4c7acc19909bbd1abb28a05f38ff14cdf WatchSource:0}: Error finding container 76b435e792cf1c845b3d227906f57bd4c7acc19909bbd1abb28a05f38ff14cdf: Status 404 returned error can't find the container with id 76b435e792cf1c845b3d227906f57bd4c7acc19909bbd1abb28a05f38ff14cdf
Sep 11 04:23:18.626372 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:18.626353    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 04:23:19.395513 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:19.395474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6" event={"ID":"6822a8d4-3241-4906-bff8-59f14d8e8818","Type":"ContainerStarted","Data":"e02678c8ec7a64742f97cb4385702e76d1d7bf6e8bcefdfd49a06566e42c1bad"}
Sep 11 04:23:19.395881 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:19.395529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6" event={"ID":"6822a8d4-3241-4906-bff8-59f14d8e8818","Type":"ContainerStarted","Data":"76b435e792cf1c845b3d227906f57bd4c7acc19909bbd1abb28a05f38ff14cdf"}
Sep 11 04:23:19.395881 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:19.395573    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:19.412416 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:19.412372    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6" podStartSLOduration=1.103232359 podStartE2EDuration="1.412357737s" totalImagesPullingTime="309.125378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:23:19.411097034 +0000 UTC m=+1997.277116763" watchObservedRunningTime="2026-09-11 04:23:19.412357737 +0000 UTC m=+1997.278377471"
Sep 11 04:23:24.674306 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:24.674274    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:23:25.406988 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:25.406960    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c9d99c6fc-drlr6"
Sep 11 04:23:25.453823 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:25.453794    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"]
Sep 11 04:23:25.454046 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:25.454021    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4" podUID="79b25eae-b255-4c81-8431-dc1cfa4b269c" containerName="maas-api" containerID="cri-o://0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7" gracePeriod=30
Sep 11 04:23:26.694542 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.694519    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 04:23:26.798892 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.798817    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6rq9\" (UniqueName: \"kubernetes.io/projected/79b25eae-b255-4c81-8431-dc1cfa4b269c-kube-api-access-x6rq9\") pod \"79b25eae-b255-4c81-8431-dc1cfa4b269c\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") "
Sep 11 04:23:26.798892 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.798845    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls\") pod \"79b25eae-b255-4c81-8431-dc1cfa4b269c\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") "
Sep 11 04:23:26.798892 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.798887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25eae-b255-4c81-8431-dc1cfa4b269c-openshift-service-ca\") pod \"79b25eae-b255-4c81-8431-dc1cfa4b269c\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") "
Sep 11 04:23:26.799103 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.798909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-metrics-certs\") pod \"79b25eae-b255-4c81-8431-dc1cfa4b269c\" (UID: \"79b25eae-b255-4c81-8431-dc1cfa4b269c\") "
Sep 11 04:23:26.799249 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.799222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b25eae-b255-4c81-8431-dc1cfa4b269c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79b25eae-b255-4c81-8431-dc1cfa4b269c" (UID: "79b25eae-b255-4c81-8431-dc1cfa4b269c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 04:23:26.800980 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.800951    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79b25eae-b255-4c81-8431-dc1cfa4b269c" (UID: "79b25eae-b255-4c81-8431-dc1cfa4b269c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:23:26.801087 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.801026    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b25eae-b255-4c81-8431-dc1cfa4b269c-kube-api-access-x6rq9" (OuterVolumeSpecName: "kube-api-access-x6rq9") pod "79b25eae-b255-4c81-8431-dc1cfa4b269c" (UID: "79b25eae-b255-4c81-8431-dc1cfa4b269c"). InnerVolumeSpecName "kube-api-access-x6rq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 04:23:26.801087 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.801031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79b25eae-b255-4c81-8431-dc1cfa4b269c" (UID: "79b25eae-b255-4c81-8431-dc1cfa4b269c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 04:23:26.899867 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.899840    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6rq9\" (UniqueName: \"kubernetes.io/projected/79b25eae-b255-4c81-8431-dc1cfa4b269c-kube-api-access-x6rq9\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:26.899867 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.899864    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-maas-api-tls\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:26.900039 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.899878    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b25eae-b255-4c81-8431-dc1cfa4b269c-openshift-service-ca\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:26.900039 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:26.899887    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b25eae-b255-4c81-8431-dc1cfa4b269c-metrics-certs\") on node \"ip-10-0-131-122.ec2.internal\" DevicePath \"\""
Sep 11 04:23:27.432468 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.432439    2566 generic.go:356] "Generic (PLEG): container finished" podID="79b25eae-b255-4c81-8431-dc1cfa4b269c" containerID="0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7" exitCode=0
Sep 11 04:23:27.432629 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.432484    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"
Sep 11 04:23:27.432629 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.432524    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4" event={"ID":"79b25eae-b255-4c81-8431-dc1cfa4b269c","Type":"ContainerDied","Data":"0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7"}
Sep 11 04:23:27.432629 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.432572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4" event={"ID":"79b25eae-b255-4c81-8431-dc1cfa4b269c","Type":"ContainerDied","Data":"be17bcb824d71d1917fe0ac8350a7f3b7cf527229e080c718d78f96bcabb4ea3"}
Sep 11 04:23:27.432629 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.432594    2566 scope.go:117] "RemoveContainer" containerID="0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7"
Sep 11 04:23:27.441136 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.441121    2566 scope.go:117] "RemoveContainer" containerID="0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7"
Sep 11 04:23:27.441351 ip-10-0-131-122 kubenswrapper[2566]: E0911 04:23:27.441333    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7\": container with ID starting with 0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7 not found: ID does not exist" containerID="0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7"
Sep 11 04:23:27.441401 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.441359    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7"} err="failed to get container status \"0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7\": rpc error: code = NotFound desc = could not find container \"0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7\": container with ID starting with 0bf187c48fbd39a69f580162625004bcd32361f35cdb468ddcc03d7c0664f7c7 not found: ID does not exist"
Sep 11 04:23:27.457169 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.457150    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"]
Sep 11 04:23:27.460695 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:27.460670    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-cfc75856f-h8kk4"]
Sep 11 04:23:28.612063 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:28.612031    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b25eae-b255-4c81-8431-dc1cfa4b269c" path="/var/lib/kubelet/pods/79b25eae-b255-4c81-8431-dc1cfa4b269c/volumes"
Sep 11 04:23:30.679425 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:30.679373    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:23:37.477127 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:37.477091    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-87jlk"]
Sep 11 04:23:45.953322 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:45.953297    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9gt2l_16b4f397-7fdf-4223-9c6a-3100c148db24/manager/0.log"
Sep 11 04:23:46.073248 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:46.073219    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t6kbc_99aabf96-b1fe-4c82-af00-e9e034e69e14/manager/0.log"
Sep 11 04:23:46.202526 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:46.202494    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-848cd9bd7-ccx4s_cd990481-d6f9-4876-8a77-57f4fa010265/manager/0.log"
Sep 11 04:23:46.324284 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:46.324261    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4kqzt_46b23d79-7b2e-4781-aadb-8abb5b8e224c/server/0.log"
Sep 11 04:23:46.738616 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:46.738592    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97cf4878d-bp9vs_7dbdf4d6-5fcf-4ade-adc5-78f535593c2c/copy-manifests/0.log"
Sep 11 04:23:46.750419 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:46.750382    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97cf4878d-bp9vs_7dbdf4d6-5fcf-4ade-adc5-78f535593c2c/manager/0.log"
Sep 11 04:23:47.341742 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:47.341716    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c9d99c6fc-drlr6_6822a8d4-3241-4906-bff8-59f14d8e8818/maas-api/0.log"
Sep 11 04:23:47.576208 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:47.576179    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818335-nn6bb_0c742d05-f1db-4f31-8a5f-e491eb01e4cd/cleanup/0.log"
Sep 11 04:23:48.648499 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:48.648471    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fpjqt_5282db95-0b76-4416-9840-34d992e4ef3a/manager/0.log"
Sep 11 04:23:49.133501 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:49.133478    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-87jlk_84a96ab1-cd48-409d-90bc-7af13fbcea18/limitador/0.log"
Sep 11 04:23:49.277021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:49.276992    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-c526b_4d6a69fa-0ff7-45df-ae28-48c109de7e9d/manager/0.log"
Sep 11 04:23:50.016025 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:50.015995    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5f7bb976-c5lcl_bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d/kube-auth-proxy/0.log"
Sep 11 04:23:50.140948 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:50.140921    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nmjcb_2028fff4-7d56-4cee-be2c-0e1e054793a5/istio-proxy/1.log"
Sep 11 04:23:50.365358 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:50.365284    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l5kqq_3338e8c2-2273-4099-aed2-4a55c00c007e/payload-processing/0.log"
Sep 11 04:23:51.244021 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:51.243988    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nw4r6_e555900e-7f9f-4d72-9d6d-abe296285ccf/main/0.log"
Sep 11 04:23:51.515219 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:51.515142    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ctbnw_d2158593-b29c-496d-a333-01027d5f3957/main/0.log"
Sep 11 04:23:51.631110 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:51.631083    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-2ldk2_e235ecec-31f0-4974-add9-7bb428973b8c/main/0.log"
Sep 11 04:23:57.677471 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.677441    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vb5nd/must-gather-6fpcr"]
Sep 11 04:23:57.677854 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.677751    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b25eae-b255-4c81-8431-dc1cfa4b269c" containerName="maas-api"
Sep 11 04:23:57.677854 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.677761    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b25eae-b255-4c81-8431-dc1cfa4b269c" containerName="maas-api"
Sep 11 04:23:57.677854 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.677828    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b25eae-b255-4c81-8431-dc1cfa4b269c" containerName="maas-api"
Sep 11 04:23:57.680810 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.680788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vb5nd/must-gather-6fpcr"
Sep 11 04:23:57.685108 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.685079    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vb5nd\"/\"default-dockercfg-pfqxn\""
Sep 11 04:23:57.685217 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.685203    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vb5nd\"/\"openshift-service-ca.crt\""
Sep 11 04:23:57.686015 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.686001    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vb5nd\"/\"kube-root-ca.crt\""
Sep 11 04:23:57.689941 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.689922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vb5nd/must-gather-6fpcr"]
Sep 11 04:23:57.748004 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.747985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcr7p\" (UniqueName: \"kubernetes.io/projected/b00bcc75-7e26-4d0f-a4c5-d287f88b77d6-kube-api-access-kcr7p\") pod \"must-gather-6fpcr\" (UID: \"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6\") " pod="openshift-must-gather-vb5nd/must-gather-6fpcr"
Sep 11 04:23:57.748106 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.748024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b00bcc75-7e26-4d0f-a4c5-d287f88b77d6-must-gather-output\") pod \"must-gather-6fpcr\" (UID: \"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6\") " pod="openshift-must-gather-vb5nd/must-gather-6fpcr"
Sep 11 04:23:57.849240 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.849216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcr7p\" (UniqueName: \"kubernetes.io/projected/b00bcc75-7e26-4d0f-a4c5-d287f88b77d6-kube-api-access-kcr7p\") pod \"must-gather-6fpcr\" (UID: \"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6\") " pod="openshift-must-gather-vb5nd/must-gather-6fpcr"
Sep 11 04:23:57.849329 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.849250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b00bcc75-7e26-4d0f-a4c5-d287f88b77d6-must-gather-output\") pod \"must-gather-6fpcr\" (UID: \"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6\") " pod="openshift-must-gather-vb5nd/must-gather-6fpcr"
Sep 11 04:23:57.849523 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.849510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b00bcc75-7e26-4d0f-a4c5-d287f88b77d6-must-gather-output\") pod \"must-gather-6fpcr\" (UID: \"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6\") " pod="openshift-must-gather-vb5nd/must-gather-6fpcr"
Sep 11 04:23:57.857091 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.857068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcr7p\" (UniqueName: \"kubernetes.io/projected/b00bcc75-7e26-4d0f-a4c5-d287f88b77d6-kube-api-access-kcr7p\") pod \"must-gather-6fpcr\" (UID: \"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6\") " pod="openshift-must-gather-vb5nd/must-gather-6fpcr"
Sep 11 04:23:57.990035 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:57.990017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vb5nd/must-gather-6fpcr"
Sep 11 04:23:58.258516 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:58.258446    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vb5nd/must-gather-6fpcr"]
Sep 11 04:23:58.261427 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:23:58.261374    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb00bcc75_7e26_4d0f_a4c5_d287f88b77d6.slice/crio-4a7d9c52c16f59c58f67590a3140ff014560e3dba755e7a538889313c8b693bb WatchSource:0}: Error finding container 4a7d9c52c16f59c58f67590a3140ff014560e3dba755e7a538889313c8b693bb: Status 404 returned error can't find the container with id 4a7d9c52c16f59c58f67590a3140ff014560e3dba755e7a538889313c8b693bb
Sep 11 04:23:58.568317 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:58.568237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vb5nd/must-gather-6fpcr" event={"ID":"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6","Type":"ContainerStarted","Data":"4a7d9c52c16f59c58f67590a3140ff014560e3dba755e7a538889313c8b693bb"}
Sep 11 04:23:59.577865 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:59.577778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vb5nd/must-gather-6fpcr" event={"ID":"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6","Type":"ContainerStarted","Data":"b82cdb86f9c3b590660ed9c47c600c1cf8868560a7cf78b9787fc10745a8543e"}
Sep 11 04:23:59.577865 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:59.577831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vb5nd/must-gather-6fpcr" event={"ID":"b00bcc75-7e26-4d0f-a4c5-d287f88b77d6","Type":"ContainerStarted","Data":"94604b585744e8ab95a760dae51366412fa6f06c17c61d7a5f5c197a53f37b1d"}
Sep 11 04:23:59.595123 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:23:59.595072    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vb5nd/must-gather-6fpcr" podStartSLOduration=1.737490381 podStartE2EDuration="2.595052933s" totalImagesPullingTime="857.562552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:23:59.593550331 +0000 UTC m=+2037.459570078" watchObservedRunningTime="2026-09-11 04:23:59.595052933 +0000 UTC m=+2037.461072667"
Sep 11 04:24:00.660949 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:00.660904    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-frxxs_37437296-da5a-4a64-8dca-a89fa0263d79/global-pull-secret-syncer/0.log"
Sep 11 04:24:00.829556 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:00.829522    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xh7zf_3cd262c2-7fea-463a-9c9d-84134976390d/konnectivity-agent/0.log"
Sep 11 04:24:00.847908 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:00.847880    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-122.ec2.internal_147e739407ba5f1c7ea961026966fc31/haproxy/0.log"
Sep 11 04:24:04.996548 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:04.996506    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fpjqt_5282db95-0b76-4416-9840-34d992e4ef3a/manager/0.log"
Sep 11 04:24:05.183954 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:05.183923    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-87jlk_84a96ab1-cd48-409d-90bc-7af13fbcea18/limitador/0.log"
Sep 11 04:24:05.278534 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:05.278451    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-c526b_4d6a69fa-0ff7-45df-ae28-48c109de7e9d/manager/0.log"
Sep 11 04:24:07.325755 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:07.325720    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qcczh_7577d019-76b8-4fdc-bfc2-c57d03cad2c6/node-exporter/0.log"
Sep 11 04:24:07.384180 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:07.384118    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qcczh_7577d019-76b8-4fdc-bfc2-c57d03cad2c6/kube-rbac-proxy/0.log"
Sep 11 04:24:07.429040 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:07.429006    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qcczh_7577d019-76b8-4fdc-bfc2-c57d03cad2c6/init-textfile/0.log"
Sep 11 04:24:10.201708 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.201674    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"]
Sep 11 04:24:10.210034 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.209998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.216626 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.216602    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"]
Sep 11 04:24:10.280843 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.280802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nv7s\" (UniqueName: \"kubernetes.io/projected/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-kube-api-access-6nv7s\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.281033 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.280891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-lib-modules\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.281033 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.280960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-podres\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.281033 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.280986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-proc\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.281033 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.281011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-sys\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382174 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nv7s\" (UniqueName: \"kubernetes.io/projected/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-kube-api-access-6nv7s\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382512 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-lib-modules\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382682 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-podres\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382769 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-lib-modules\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382769 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-proc\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382769 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-sys\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382911 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-proc\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382911 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-sys\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.382911 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.382838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-podres\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.391329 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.391308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nv7s\" (UniqueName: \"kubernetes.io/projected/8cc19bd6-acab-43a4-bf02-bcd09dabb6c8-kube-api-access-6nv7s\") pod \"perf-node-gather-daemonset-85mxt\" (UID: \"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8\") " pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.527618 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.527577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:10.808059 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:10.808034    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"]
Sep 11 04:24:10.810805 ip-10-0-131-122 kubenswrapper[2566]: W0911 04:24:10.810775    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8cc19bd6_acab_43a4_bf02_bcd09dabb6c8.slice/crio-1638dc7a485741b020d99ed5a1d6d546e4868a3c75e36c6e735e4e1ec3fe8979 WatchSource:0}: Error finding container 1638dc7a485741b020d99ed5a1d6d546e4868a3c75e36c6e735e4e1ec3fe8979: Status 404 returned error can't find the container with id 1638dc7a485741b020d99ed5a1d6d546e4868a3c75e36c6e735e4e1ec3fe8979
Sep 11 04:24:11.673606 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:11.673570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt" event={"ID":"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8","Type":"ContainerStarted","Data":"6e5ab169370fbf2cbe3e905d60d2f7fe404b7450905068b30bc946a8e9ca4eb5"}
Sep 11 04:24:11.674175 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:11.674139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt" event={"ID":"8cc19bd6-acab-43a4-bf02-bcd09dabb6c8","Type":"ContainerStarted","Data":"1638dc7a485741b020d99ed5a1d6d546e4868a3c75e36c6e735e4e1ec3fe8979"}
Sep 11 04:24:11.675041 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:11.675020    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:11.692399 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:11.692348    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt" podStartSLOduration=1.69233115 podStartE2EDuration="1.69233115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 04:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:24:11.690943534 +0000 UTC m=+2049.556963270" watchObservedRunningTime="2026-09-11 04:24:11.69233115 +0000 UTC m=+2049.558350886"
Sep 11 04:24:12.624535 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:12.624499    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2trdb_6e3f4a25-3d98-48b6-9026-045c9e3df008/dns/0.log"
Sep 11 04:24:12.647078 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:12.647049    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2trdb_6e3f4a25-3d98-48b6-9026-045c9e3df008/kube-rbac-proxy/0.log"
Sep 11 04:24:12.765037 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:12.765012    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2d7rk_82a364ce-5981-412b-9cf0-65da78af29d9/dns-node-resolver/0.log"
Sep 11 04:24:13.322266 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:13.322232    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86d879bdf9-p9lqk_f70e391d-9436-4989-b9ed-992782a0b750/registry/0.log"
Sep 11 04:24:13.372398 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:13.372364    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6lvvf_cdc800d9-e949-4c5e-9804-1f74c6b507b9/node-ca/0.log"
Sep 11 04:24:14.603455 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:14.603430    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5f7bb976-c5lcl_bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d/kube-auth-proxy/0.log"
Sep 11 04:24:14.631136 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:14.631107    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nmjcb_2028fff4-7d56-4cee-be2c-0e1e054793a5/istio-proxy/0.log"
Sep 11 04:24:14.642903 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:14.642871    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nmjcb_2028fff4-7d56-4cee-be2c-0e1e054793a5/istio-proxy/1.log"
Sep 11 04:24:14.713998 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:14.713969    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l5kqq_3338e8c2-2273-4099-aed2-4a55c00c007e/payload-processing/0.log"
Sep 11 04:24:15.574204 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:15.574180    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-btpdt_4eeec831-824a-4089-87ff-a203bf1d9a87/serve-healthcheck-canary/0.log"
Sep 11 04:24:16.190392 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:16.190365    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqk7x_7015feea-77c2-4835-a93b-25fce799b674/kube-rbac-proxy/0.log"
Sep 11 04:24:16.210148 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:16.210124    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqk7x_7015feea-77c2-4835-a93b-25fce799b674/exporter/0.log"
Sep 11 04:24:16.231199 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:16.231173    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wqk7x_7015feea-77c2-4835-a93b-25fce799b674/extractor/0.log"
Sep 11 04:24:18.247524 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:18.247465    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9gt2l_16b4f397-7fdf-4223-9c6a-3100c148db24/manager/0.log"
Sep 11 04:24:18.475171 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:18.475133    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-t6kbc_99aabf96-b1fe-4c82-af00-e9e034e69e14/manager/0.log"
Sep 11 04:24:18.529064 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:18.528980    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-848cd9bd7-ccx4s_cd990481-d6f9-4876-8a77-57f4fa010265/manager/0.log"
Sep 11 04:24:18.548306 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:18.548281    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4kqzt_46b23d79-7b2e-4781-aadb-8abb5b8e224c/server/0.log"
Sep 11 04:24:18.696580 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:18.696553    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vb5nd/perf-node-gather-daemonset-85mxt"
Sep 11 04:24:18.762445 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:18.762388    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97cf4878d-bp9vs_7dbdf4d6-5fcf-4ade-adc5-78f535593c2c/manager/0.log"
Sep 11 04:24:18.778664 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:18.778640    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97cf4878d-bp9vs_7dbdf4d6-5fcf-4ade-adc5-78f535593c2c/copy-manifests/0.log"
Sep 11 04:24:20.050151 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:20.050111    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ftqtb_436c563e-bce9-4235-8ba4-c1ef14eee54a/openshift-lws-operator/0.log"
Sep 11 04:24:20.189698 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:20.189619    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5f7bb976-c5lcl_bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d/kube-auth-proxy/0.log"
Sep 11 04:24:20.211889 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:20.211856    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nmjcb_2028fff4-7d56-4cee-be2c-0e1e054793a5/istio-proxy/0.log"
Sep 11 04:24:20.223309 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:20.223285    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nmjcb_2028fff4-7d56-4cee-be2c-0e1e054793a5/istio-proxy/1.log"
Sep 11 04:24:20.274794 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:20.274764    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l5kqq_3338e8c2-2273-4099-aed2-4a55c00c007e/payload-processing/0.log"
Sep 11 04:24:22.092763 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:22.092735    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5f7bb976-c5lcl_bd285f9c-6e7a-4461-8ac7-dfdcb89cb22d/kube-auth-proxy/0.log"
Sep 11 04:24:22.110765 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:22.110743    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nmjcb_2028fff4-7d56-4cee-be2c-0e1e054793a5/istio-proxy/0.log"
Sep 11 04:24:22.120628 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:22.120608    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nmjcb_2028fff4-7d56-4cee-be2c-0e1e054793a5/istio-proxy/1.log"
Sep 11 04:24:22.179173 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:22.179100    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l5kqq_3338e8c2-2273-4099-aed2-4a55c00c007e/payload-processing/0.log"
Sep 11 04:24:25.186085 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:25.186014    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-6bgsn_84d18f9d-c86c-45fe-9fd9-53883ccc5eb7/migrator/0.log"
Sep 11 04:24:25.209542 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:25.209520    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-6bgsn_84d18f9d-c86c-45fe-9fd9-53883ccc5eb7/graceful-termination/0.log"
Sep 11 04:24:26.501314 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:26.501285    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2mf6_b98300df-7e61-45ad-87d0-b0185bb00771/kube-multus-additional-cni-plugins/0.log"
Sep 11 04:24:26.523333 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:26.523311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2mf6_b98300df-7e61-45ad-87d0-b0185bb00771/egress-router-binary-copy/0.log"
Sep 11 04:24:26.544240 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:26.544216    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2mf6_b98300df-7e61-45ad-87d0-b0185bb00771/cni-plugins/0.log"
Sep 11 04:24:26.566006 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:26.565987    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2mf6_b98300df-7e61-45ad-87d0-b0185bb00771/bond-cni-plugin/0.log"
Sep 11 04:24:26.598742 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:26.598725    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2mf6_b98300df-7e61-45ad-87d0-b0185bb00771/routeoverride-cni/0.log"
Sep 11 04:24:26.618929 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:26.618911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2mf6_b98300df-7e61-45ad-87d0-b0185bb00771/whereabouts-cni-bincopy/0.log"
Sep 11 04:24:26.639064 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:26.639048    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2mf6_b98300df-7e61-45ad-87d0-b0185bb00771/whereabouts-cni/0.log"
Sep 11 04:24:26.997264 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:26.997234    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c4lfh_e1b6be1d-bba4-48f0-900a-a36c80597729/kube-multus/0.log"
Sep 11 04:24:27.158355 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:27.158322    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k57hw_c6f66daa-4ff1-45b4-93d6-cc8de58802b3/network-metrics-daemon/0.log"
Sep 11 04:24:27.174892 ip-10-0-131-122 kubenswrapper[2566]: I0911 04:24:27.174869    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k57hw_c6f66daa-4ff1-45b4-93d6-cc8de58802b3/kube-rbac-proxy/0.log"